One of the most important parts of data analysis is getting data (amazing, right?). But not just any data, a huge amount of quality data. Usually, you can find it through APIs.
The .NETCore console applications allow to quickly create executables to be deployed under Linux. In our case, we use it to periodically retrieve data from an API to populate our own database.
So far, this is for me one of the fastest ways to call an HTTP client. It does not depend on Nginx/Apache configuration or on a third-party library. You just need to install .NET Core runtime on your server.
If you need some APIs to follow this tutorial, I hope you will find your happiness from this repository (it's free to use):
public-apisCreate a .NET Core console
To start, let's create a .NET Core console application.
It will create a project with one file called Program.cs. The command will always start from the function Main inside.
HTTP client to request API
Before developing the main function, we start by coding the GetDataFromAPI function. Its goal is to create an HTTP client to request the API data.
Most of the time, the way to get data from an API is the same: generate the final URL by concatenating the domain URL with filtering parameters. The user API key can also be added inside request headers. That's it.
Here is an example from a function to get Binance data. You can take inspiration from the following code :
#Program.cs
private static string GetDataFromAPI()
{
var parameters = new Dictionary<string, string>() {
{ "PARAM1", "PARAM1VALUE" },
{ "PARAM2", "PARAM2VALUE" }
};
using (HttpClient client = new HttpClient())
{
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
client.DefaultRequestHeaders.Add("APIKEYPARAM", "APIKEY");
var baseUrl = "https://URLAPI.com";
var url = baseUrl + "?" + string.Join("&", parameters.Select(x => string.Format("{0}={1}", x.Key, x.Value)));
var task = client.GetAsync(url);
task.Wait();
if (!task.Result.IsSuccessStatusCode)
{
throw new Exception(task.Result.StatusCode.ToString());
}
var readTask = task.Result.Content.ReadAsStringAsync();
readTask.Wait();
return readTask.Result;
}
}It generates an URL (looks like this one: https://api.binance.com/api/v3/klines?symbol=BNBBTC&interval=1d) and returns results as a string (since API returns JSON).
I won't go further in the logic because it is not the purpose of this article. At this point, you need to develop the storage part to save the results.
We also modify the Main function with just enough logs to know if the command succeeded or not.
#Program.cs
static void Main(string[] args)
{
try
{
Console.WriteLine("API import starting...");
var results = GetDataFromAPI();
//SaveResults();
Console.WriteLine("API import ended successfully");
}
catch (Exception ex)
{
Console.WriteLine("Error : " + ex.Message);
}
}This is it. We have a strict minimum basis to query an API.
Make your application more flexible
At this point, we missed 2 important things :
- Arguments to change application's parameters
- Logs to overview what happens
Manage command arguments
The first step is to accept arguments from the command line, such as our API Key. This way, we will be able to launch multiple applications at the same time if we need to parallelize.
Let's take an example. In the command tail -n 1 /path, tail is the command, -n is an option and 1 is the value of the n option.
Since we are creating a command, we can do exactly the same thing in our application by defining options.
public class CommandLineOptions
{
[Option('a', "apikey", Required = true, HelpText = "API Key")]
public string ApiKey
{
get; set;
}
//.... We can add as much as we need
}If the method Option() is underline red, you have to import the package CommandLineParser. You can install it by right-clicking on Option() or by using Nuget > CommandLineParser > Install.
Log command returns
Instead of returning only the error message, we add the code and the stack trace to make debugging easier. Feel free to improve this logging.
public class ReturnObject
{
public int Code { get; set; }
public string ErrorMessage { get; set; }
public string StackTrace { get; set; }
}Final Main function
By including command line arguments and logs, we obtain :
#Program.cs
static void Main(string[] args)
{
// This method is called to manage arguments (included in CommandLineParser library)
Parser.Default.ParseArguments<CommandLineOptions>(args).WithParsed(o =>
{
var objectReturn = new ReturnObject() { Error = 0 };
try
{
Console.WriteLine("API import starting...");
// You can pass your options as parameters
var results = GetDataFromAPI(o.ApiKey);
// SaveResults();
Console.WriteLine("API import ended successfully");
}
catch (Exception ex)
{
objectReturn.Error = -1;
objectReturn.ErrorMessage = ex.Message;
objectReturn.StackTrace = ex.StackTrace;
Console.Write(JsonConvert.SerializeObject(objectReturn));
}
});
}How to test
Visual Studio allows us to debug our command before deploying it on our server.
We simulate the command line in the console with specific arguments on project properties. We can add the -a option by default when we start debugging.
It instantiates the console application and starts the Main function with our argument MYAPIKEY.
You will be able to locally validate your code before going further.
Deploy the application on Linux
The last step is to publish the command to deploy it on your server. You will find the "Publish" option by left-clicking on the project.
Define the parameters according to the configuration of your server. In this example, we have a Linux server with the 3.1 runtime installed. The publication will generate the executable and all the dependencies.
Send all files on your server with an FTP client (like Filezilla) and change permission on the command named YourConsoleAppName.
You can test our console app by using the following command : ./YourConsoleAppName -a "MyAPIKEY"
Add to Crontab
To be sure to get the latest data, we add the command on the Linux crontab tool.
For example 1 * * * * ./update.sh -a "MYAPIKEY"
In this case, it will run our command every hour and 1 minute. I prefer to leave a 1-minute delay to make sure I get the first-time value of each hour. See the documentation to change the settings.
Once you set the time-based scheduler, copy the line in the file opened by the command sudo crontab -e.
Recap
A summary of what you have just learned:
- Create a .NETCore command and request HTTP clients
- Manage arguments/logs and how to test it
- Deploy application on Linux in order to create a CRON task to get the API data permanently
Enjoy!
Join the conversation.