Problem formulation
You have a dataset in which each observation is an impression of a banner ad. You have a variable indicating success (say, click or conversion) and a lot of additional information (features), all of which are coded as binary: browser type, host, URL, placement, banner format, banner id, banner keywords, hosts the viewer has seen so far, how many times the viewer has seen particular banners, when did s/he see these banners, how many times did s/he click on which banners, how did s/he behave while shopping online, current date and time, geoid information, and so on.
The question is which features increase the chance of success and which decrease it? This is an important question if you want to allocate your advertising resources efficiently. The difficulty is that the number of observations is in billions and the number of available features is in millions.
A solution
A naïve approach is to create a table which informs how many successes and how many failures occurred when feature was present or absent. Then, you can compare success ratio in absence of the feature with the success ratio in presence of the feature. If the latter is higher than the former, then the feature indicates higher probability of success.
This approach is similar to calculating simple correlation between the feature and the success indicator. And thus, it suffers from endogeneity. If a combination of two features often occurs together, say a particular host and a particular banner, and both of them seem to have high correlation with the success indicator, you do not really know whether this is the banner that drives success, the host, or both.
In order to separate the effects of features, you need to calculate partial correlation, conditional on other features, rather than simple correlation. The straightforward way to do it is to perform an ordinary least squares regression on the data. Unfortunately, there exists no software that could handle amounts of data you have. Even if you limit the dataset to most common features – say top 5000 – you still end up with several terabytes of data to be processed by the regression algorithm. To focus attention, let us say that we need a way to perform a regression on n = 4 billion observations and k = 10 thousand features. If each variable takes up 4 bytes, the amount of memory required to perform such analysis equals nearly 160 terabytes.
Typically, linear least squares models are fit using orthogonal decomposition of the data matrix. For example, R package uses QR decomposition. One can use also singular value decomposition. Unfortunately, these methods require all data to be kept in memory and have algorithmic complexity of O(nk2).
Alternatively, one can calculate Gram matrix. This has algorithmic complexity of O(nk2) which can be reduced to O(np2) if the data are sparse (where p is the quadratic mean number of features per observation) and very easily parallelized. Another advantage is that memory requirement for calculating Gram matrix are O(k2) only and for k = 10000 the exact amount of RAM required to keep Gram matrix would be just under 200 MB (keep in mind that Gram matrix is symmetric). The only problem here is that to calculate regression coefficients, it is necessary to invert the calculated Gram matrix (which is often discouraged due to inferior numerical stability and takes O(k3)). The viability of this solution depends thus on whether it is possible to do it with satisfactory numerical accuracy. As it turns out, it is.
Note that popular machine learning engines like Vowpal Wabbit are not of much use in this situation. Machine learning is usually concentrated on prediction, rather than accurate estimation of model parameters. Engines like VW in principle are less accurate than OLS. They allow multi-collinearity of variables which in turn forces user to perform separate data analysis in order to eliminate it in the first place. Finally, they do not allow for standard statistical inference with the model parameters.
Preliminaries
The plan was to create a C++ class able to do all operations necessary for this regression. The data were stored on a remote Linux server using Hadoop. I was planning to develop and debug my solution using Microsoft Visual Studio 2015 on my Windows 7 64-bit Dell computer (i7-4790 @ 3.6 GHz with 16 GB RAM) and then to port it to its final destination.
There were four initial things I had to take care of: (1) a way of measuring code performance, (2) a way of measuring numerical accuracy of matrix inversion, (3) C++ libraries for inverting matrices, and (4) a strategy for verifying accuracy of the entire algorithm.
Boy, was it hard to find a good way to precisely measure code execution time on Windows. Unfortunately, the usually recommended GetTickCount() Windows API function relies on the 55 Hz clock and thus has a resolution of around 18 milliseconds. Fortunately, I eventually found out about the QueryPerformanceCounter() function, whose resolution is much better.
Next, I decided to use the following measure for numerical precision of matrix inversion. Let us say that you need to invert matrix A. You use an inversion algorithm on it which generates matrix B. If matrix B is a perfect inverse of A, then AB = I, where I is the identity matrix. Hence, I calculate matrix C = AB – I. Then, I find the element of matrix C that has the highest absolute value and call it r. This is my measure of numerical precision. In the world of infinite precision, r = 0. In the real world r < 1e-16 is perfect (I use double – a 64 bit floating point type for my calculations). r < 1e-5 is still acceptable. Otherwise there are reasons to worry.
With tools for measuring performance and accuracy, I was able to start testing libraries. I initially turned to Eigen which was very easy to install and use with my Visual Studio. Eigen uses LU decomposition for calculating matrix inverse and was satisfying in terms of speed and reliability – up to the point when I tried to invert a 7000x7000 matrix. Eigen kept on crashing and I could not figure out why. The second option was thus Armadillo. Armadillo did not have the same problems and worked well with bigger matrices all the way up to 10000.
As it turns out, Armadillo can take advantage of the fact that Gram matrix is symmetric and positive-definite. The inversion is done by means of Cholesky decomposition and after a few experiments I realized that it is not only faster but also numerically more reliable than LU-based method. I was able to invert a 10001x10001 matrix in 283 seconds (in a single thread) with r = 3.13e-14. The irony is that both Cholesky decomposition and matrix multiplication work in O(k3) but the latter is over twice as slow, so it takes much more time to check numerical precision than to perform actual inversion.
Finally, I designed a data generating process to test whether least squares algorithm of my design can recover parameters used to generate the data. Essentially, I created 10001 variables xi for i=0, 1, 2, …, 10000. x0 = 1, always. For i>0 we have P(xi = 1) = 1/(3+i) = 1 – P(xi = 0). Then, I created a vector of parameters bi. b0 = 0.0015 and for any non-negative integer j, b4j+1 = 0.0001, b4j+2 = 0.0002, b4j+3 = 0.0003, and b4j+4 = -0.00005. Finally, P(y = 1) = x * b, where * indicates dot product. This is a typical linear probability model.
Using the formula above I generated 4 billion observations (it took 11 days on 4 out of 8 cores of my Windows machine) and fed them into the regression algorithm. The algorithm was able to recover vector b with the expected convergence rate. Note that by design the aforementioned data generating process creates variables that are independently distributed. I thus had to tweak this and that to see whether the algorithm could handle correlated features as well as to investigate the bias (see more about that in the last section).
Statistical inference
The question of how to recover model parameters from the data is simple. In addition to the Gram matrix, you need a success count vector. The i-th element in this vector indicates how many successes were there when i-th feature was present. Calculating this vector is at most O(np) in time and requires O(k) memory (note that none of the operations involved in calculating Gram matrix and success count vector are floating point operations – this is all integer arithmetic since we operate on binary variables only; thus both Gram matrix and success count vector are calculated with perfect numerical precision). Once you have them both, you need to invert the Gram matrix and multiply it by the success count vector. The resulting vector contains estimated model parameters.
However, getting standard errors of the estimated coefficients is a bit more complicated. Typically, we would use diagonal elements of the inverted Gram matrix and multiply them by standard deviation of the residuals. The problem is that calculating residuals requires going through all observations all over again. This not only increases the execution time. It poses a major technical difficulty as it requires the dataset to be invariant for the duration of the algorithm execution (which is assumed to be at least several hours). To fix this, one would have to tinker with the data flow in the entire system which can greatly inflate project’s costs.
Fortunately, there is trick that can rescue us here. Instead of quadratic mean of residuals, one can use standard deviation of the success variable. Note that the latter must be greater than the former: the former is the quadratic mean of residuals for the entire model and the latter is the quadratic mean of residuals for the model with a constant only. This guarantees that the standard errors will be overestimated which is much better than having them underestimated or all over the place. Moreover, for small average success ratio, the two will be close. In fact, it is easy to show that under some plausible conditions as the average success ratio goes to zero, the two are the same in the limit. And for banner impressions, the average success ratio (e.g. CTR) is, no doubt, small.
No amount of theoretical divagations can replace an empirical test. It is thus necessary to check ex post whether statistical inference using the above simplifications is indeed valid. To do that, I estimate a number of models (keep in mind that I have 10000 variables) and check how frequently the estimated coefficients are within the 95% confidence intervals. I expect them to be there slightly more often than 95% of the time (due to overestimation of standard errors) and indeed, this is what I find.
Finally, I cannot write a section about statistical inference without bashing p-values and t-statistics. I strongly discourage you from using them. A single number is often not enough to facilitate good judgment about the estimated coefficient. p-value typically answers a question like: “how likely is it, that the coefficient is on the opposite side of zero?” - Is this really what you want to know? The notion of statistical significance is often misleading. You can have a statistically insignificant coefficient whose confidence interval is so close to zero that any meaningful influence on the dependent variable is ruled out: you can then say that your data conclusively show that there is no influence (rather than that the data do not show that there is influence). Also, you can have a statistically significant coefficient with very high t-statistic, which is economically insignificant or economically significant but estimated very imprecisely. Thus, instead of p-values and t-statistics I suggest using confidence intervals. The question they answer is: what are the likely values of the coefficient? And this is what you actually want to know most of the time.
Data refinements
Oops. You have a nice OLS algorithm which supports valid statistical inference. You tested it with you generated data and it works fine. Now you apply it to real data and the Gram matrix does not want to invert or inverts with precision r > 1. You quickly realize that it is because the data have a lot of constant, perfectly correlated, and multicollinear variables. How to deal with that?
Sure, you can force users to limit themselves only to variables which are neither perfectly correlated nor multi-collinear. But when they are using thousands of variables, it may take a lot of effort to figure it out. Also, running an algorithm for several hours only to learn that it fails because you stuffed it with a bad variable (and it does not tell you which one is bad!) simply does not seem right. Fortunately, as it turns out, all these problems can be fixed with analysis and manipulations on the already-calculated Gram matrix.
The first refinement I suggest is dropping features that are present too few times (e.g. less than 1000). You can find them by examining diagonal entries of the Gram matrix. To drop a variable you can just delete appropriate row and column from the Gram matrix as well as corresponding entry form the success count vector. After such a delete operation, what you are left with is the same as if you did not consider the deleted variable to begin with. Clear cut.
The second refinement I suggest is to drop features with not enough variability. Based on the Gram matrix and the success count vector, it is possible to construct a variability table for every feature (the same one I described as the naïve solution at the beginning of the article). This table has two rows and two columns – rows indicate whether there was a success and columns indicate whether the feature was present. Each cell contains the number of observations. So you have the number of observations that had a feature and there was a success, a number of observations that had a feature but with no success, a number of observation without this feature but with success, and a number of observations with neither feature nor success. I drop features for which at least one of the four cells has a value lower than 10.
As we proceed with the third refinement, note that you can easily calculate correlation between any two features based on the content of Gram matrix. Just write out the formula for correlation and simplify it knowing that you are dealing with binary variables to realize that you have all information you need in the Gram matrix. This of course allows you to identify all pairs of perfectly correlated or highly correlated variables in O(k2) time. I got rid of a variable if I saw correlation whose absolute value exceeded .99 (doing, say, .95 instead of .99 did not dramatically improve speed or numerical precision of the algorithm).
But now comes a biggie. How to find features that are perfectly multicollinear? One naïve approach is to try to find all triples of such variables and test them for multicollinearity, find all quadruples, quintuples, and so on. The trouble is that finding all n-tuples can be done in time O(kn) which is a nightmare. Alternatively, you can try to invert submatrices: if you can invert a matrix made up of first p rows and columns of the original Gram matrix, but you cannot invert a matrix made up of the first p+1 rows and columns of the original, it surely indicates that the variable number p+1 causes our Gram matrix to be singular. But this solution has a complexity of O(k4) which for high k may be very cumbersome. There must be a better way.
As it turns out a better way is to perform a QR decomposition of the Gram matrix (not to confuse with QR decomposition of the data matrix as a part of the standard linear least squares algorithm). The diagonal elements of the R matrix are of interest to us – a zero indicates that a variable is causing problems and needs to be eliminated. QR decomposition generates the same results as the “invert submatrices” algorithm described above – but it runs in O(k3). And, of course, it is a good practice to check its numerical precision in a similar way we were checking numerical precision of matrix inversion algorithm.
Finally, note that you can sort the Gram matrix using its diagonal entries. I sort it descending so that features that get eliminated are always the features which occur less frequently. It is probably possible to achieve higher/lower numerical precision by sorting Gram matrix, however I have not investigated this issue extensively. I only noticed that in some instances sorting the Gram matrix ascending made the LU inversion algorithm fail (too high r) while sorting descending or not sorting did not affect the LU algorithm much.
All these operations require some effort to keep track of which variables were eliminated and why, and especially how variables in the final Gram matrix (the one undergoing inversion) map to the initial variables before the refinements. However, the results are worth the effort.
Integration and application
The task of integrating new solutions with legacy systems may be particularly hard. Fortunately, in my case, there already existed data processing routines that fed off of the same input I needed (that is a stream of observations in a sparsity supporting format – a list of “lit-up” features), as well as input generating routines that filtered original data with given observation and feature selectors.
I had a shared terminal session using Screen with people responsible for maintaining C++ code for analysis done on these data to-date. We were able to link up my class within the current setup so that users can use the same interface to run the regression that they used previously to do other type of analyses. Later on, I had to do some code debugging to account for unexpected differences in data format but ultimately everything went well.
The first real data fed to the algorithm had 1.16 billion observations and 5050 features. Calculating Gram matrix and success count vector took around 7 hours. Due to refinements, the number of features was reduced to 3104. Inverting matrix took just a few seconds, and the achieved precision was around 2e-7.
Pitfalls
In this final section I would like to discuss three potential problems that do not have easy solutions: variable cannibalization, bias, and causality.
It often happens that a number of available features refer to essentially the same thing. For example, you may have features that indicate a person who did not see this banner in past minute, 5 minutes, hour, and day. These features will be correlated and they have a clear hierarchy of implication. A user can make an attempt to run a regression using all these features expecting that the chance of success will be a decreasing function of the number of impressions. However, the effect of a viewer who has never seen the banner will not be attributed entirely to any of the aforementioned features. Instead, it will be split among them, making the estimated coefficients hard to interpret. This is the essence of cannibalization – similar variables split the effect they are supposed to pick up and therefore none of them has a coefficient it should have (please let me know if you are aware of a better term than “cannibalization”). The simple but somewhat cumbersome remedy for it is to manually avoid using features with similar meaning in one regression.
Secondly, it is widely known that linear probability model generates bias. The biased coefficients are usually closer to zero than they should be. To see why, consider a feature whose effect is to increase probability of success by 10%. However, this feature often occurs with other feature whose presence drives the probability of success to -25% (that is zero). Presence of the feature in question can at best increase the probability to -15% (that is still zero). As a result the feature in question does not affect the outcome in some sub-population due to negative predicted probability. Its estimated effect is thus smaller (closer to zero) than expected 10%.
Note that the reason why linear probability model generates biased results is not because the regression algorithm is flawed but because the model specification is flawed. The P(y = 1) = x * b model equation is incorrect if x * b is smaller than zero or bigger than one because probability by definition must be between 0 and 1. Whenever x * b is outside these bounds, the coefficients end up being biased. That is, OLS correctly estimates partial correlation between independent and dependent variables, but, due to data truncation, partial correlation is not what is needed to recover the linear probability model parameters.
The resolution of this issue may go towards assuming that model specification is correct and finding ways to alleviate bias or at least towards identifying features whose coefficients may be biased. On the other hand it may be also possible to assume that the linear probability specification is incorrect and to investigate whether partial correlation is what is really needed for the decision problems the estimates are supposed to help with. I consider solving this problem an issue separate from the main topic of this article and I leave it at that.
Finally, I would like to make a note on causality. Partial correlation, as any correlation, does not imply causation. Therefore, it may turn out that a particular feature does not have a causal effect on probability of success but instead is correlated with an omitted variable which is the true cause of the change in the observable behavior. For example, one host can have a higher conversion ratio than the other. However, the reason for that may be that the advertised product is for females only. The population of females may be much smaller for the second host even though higher fraction of them buys the product. In such case the second host is actually better at selling the product (that is it is better to direct generic traffic to the second host rather than to the first one) but this information is obscured by inability to distinguish between male and female viewers. It is thus important to remember that the regression provides us only with partial correlation rather than proofs of causality.
The issue of causality is of extreme importance when we are trying to predict effects of policy (like redirecting traffic in the example above). However, when instead of policy effects, we are interested in predictions, partial correlation seems to be a sufficient tool. For example, you may want to know whether people using Internet Explorer are more likely to click on a banner, even though you do not have the ability to influence what browser they are using. In such situations establishing causality is not necessary.
:-)
Showing posts with label IT. Show all posts
Showing posts with label IT. Show all posts
Wednesday, November 18, 2015
Saturday, July 18, 2015
Data scraping with C++
Hello.
My name is CText. I am a C++ class. I can read a text
file – usually an HTML or XML file. I can extract a lot of useful information
from it. I can find a piece of text enclosed by given strings. I can look for
it in a particular line or in the entire file. I can parse tables. I can parse
lists. I have worked with various data sources, including Amazon, Christies,
EPO, Factiva, and many more. I significantly speed up coding and
increase code reliability. Although I am probably not perfectly optimized, I am
doing my job well. I have to admit that I have been tested only with Visual C++
on Microsoft Windows. And I require an additional header file to
function. But I still hope I can be useful. The following program
illustrates my capabilities. The program
- downloads the GDP data from the World Bank website,
- converts the GDP data and saves it to a CSV file,
- finds the first line of the table in the file and displays this line, and
- retrieves the list of the countries and displays it.
Download the source code or copy and paste it:
#include <iostream>
#include <andatathresher.h>
using namespace std;
int main()
{
cout << "Hello World!" << endl;
download("http://data.worldbank.org/indicator/NY.GDP.MKTP.CD",
"file.html");
CText txt("file.html");
CCSV csv;
csv.data = txt.parsetable(0);
csv.savetofile("gdp.csv");
int line = txt.findline("<table");
cout << "The line containing first \"table\" has the number "
<< line << " and its contents is: " << endl << txt.line(line)
<< endl;
vector<string> ctrs = txt.selectiveharvest("Country name",
"</table>", "<tr", "</tr>", "<a href", ">", "<");
cout << "Here is the list of countries covered:" << endl;
for (int i = 0; i < ctrs.size() - 1; i++)
cout << ctrs[i] << "; ";
if (ctrs.size()>0) cout << ctrs[ctrs.size() - 1] << endl;
system("pause");
return 0;
}
Class members
string content
A field with the content of the file.
vector<int> separators
A field that contains positions of new line characters in
the file.
CText::CText(const string &fname)
A class constructor. Loads a file and performs its preliminary
analysis. A constructor without parameters can be called as well and file can
be loaded manually but one has to be careful about functions using lines.
bool CText::load(const string &filename)
This function loads a file with a given file name, stores
its content in the field content, and performs its primary analysis. Returns
true upon success and false otherwise.
void CText::update()
A method performing preliminary analysis of the file. It
updates the separators vector.
int CText::findline(const string &text, int pos = 0)
This function returns the number of the first line that contains
given string. It starts searching in the file from the offset pos. Returns -1
if no line is found.
int CText::findinline(string &res, int line, const
string &before, const string &after)
This function extracts a string from a line with the line
number given by line. The string that is directly after before and before after
is returned through reference res. The function returns offset of the found
string or -1 if it cannot be found.
int CText::findafter(string &res, const string
&prefix, const string &before, const string &after, int pos = 0)
This function extracts a string from a file. It starts
searching from the offset pos. It looks for the first subsequent occurrence of
prefix and then for the first subsequent occurrence of before. The string that
is directly after before and before after is returned through reference res. The
function returns offset of the found string or -1 if it cannot be found.
int CText::findbetween(string &res, const string
&before, const string &after, int pos = 0)
This function acts as findafter but with an empty string in
prefix.
int CText::geturl(string &res, const string
&pattern)
This function looks for a line that contains string pattern.
Then it searcher for the first href HTML attribute it can find and returns its
content trough reference res. The function returns offset of the found string or
-1 if it cannot be found.
string CText::line(int index)
This function returns line from the file with the line
number given by index. Lines are numbered from 0.
vector<string> CText::harvest(const string
&ldelimit, const string &udelimit, const string &prefix, const
string &before, const string &after)
This function acts as selectiveharves but with empty strings
in starter and stopper. That is, it operates on the entire file.
vector<string> CText::selectiveharvest(const
string& starter, const string& stopper, const string &ldelimit,
const string &udelimit, const string &prefix, const string &before,
const string &after)
This functions locates all strings that satisfy some
criteria and returns them as a vector. It operates only on the fragment of the
file that is after first occurrence of starter and before first subsequent occurrence
of stopper. Within this range function looks for pieces of text enclosed by
ldelimit and udelimit. Each such enclosure is supposed to produce one element
of the resulting vector. Within each enclosure, function performs procedure
similar to findafter and adds the result as an element to the resulting vector.
int CText::occurencies(const string& text)
This function counts how many times given string can be
found in the file.
vector<TDataRec> CText::parsetable(int pos)
This function parses an HTML table and returns it as a
vector of vectors of strings. It looks for the first table after offset pos. It
will not work for tables inside tables. For the definition of TDataRec type see
file anutil.h.
Class code
Download the source code or copy and paste it:
#pragma once
#include <string>
#include <vector>
#include "anutil.h"
#include <algorithm>
using namespace std;
/* INTERFACE */
class CText {
public:
string content;
vector<int> separators;
CText() {};
CText(const string &fname);
void update();
bool load(const string &filename);
int findline(const string &text, int pos);
int findinline(string &res, int line,
const string &before, const string &after);
int findafter(string &res, const string &prefix,
const string &before, const string &after, int pos);
int findbetween(string &res, const string &before,
const string &after, int pos);
int geturl(string &res, const string &pattern);
vector<string> harvest(const string &ldelimit,
const string &udelimit, const string &prefix,
const string &before, const string &after);
vector<string> selectiveharvest(const string &starter,
const string &stopper, const string &ldelimit,
const string &udelimit, const string &prefix,
const string &before, const string &after);
string line(int index);
int occurencies(const string &text);
vector<TDataRec> parsetable(int pos);
};
/* IMPLEMENTATION */
CText::CText(const string &fname)
{
load(fname);
}
void CText::update()
{
separators.clear();
for (int i=0;i<content.length();i++)
if (content[i]=='\n') separators.push_back(i);
separators.push_back(content.length());
}
bool CText::load(const string &filename)
{
ifstream infile;
infile.open(filename,ios::binary);
if (!infile.is_open())
{
content = "";
return false;
}
stringstream cont;
string line;
while (!infile.eof())
{
getline(infile,line);
cont << line << endl;
}
content = cont.str();
infile.close();
update();
return true;
}
int CText::findline(const string &text, int pos = 0)
{
if (separators.size()==0) return -1;
int k = content.find(text,pos);
if (k<0) return -1;
int i;
for (i=separators.size()-1; i>=0; i--)
if (separators[i]<k) break;
return i+1;
}
int CText::findinline(string &res, int line,
const string &before, const string &after)
{
int start = content.find(before,separators[line]);
int stop = content.find(after,start+before.length());
if ((start<0) || (stop<0)) return -1;
if (line+1<separators.size()) if ((start>separators[line+1])
|| (stop>separators[line+1])) return -1;
int pos = start + before.length();
res = content.substr(pos,stop-pos);
return pos;
}
int CText::findafter(string &res, const string &prefix,
const string &before, const string &after, int pos = 0)
{
int k1 = content.find(prefix,pos);
if (k1<0) return -1;
int k2 = content.find(before,k1+prefix.length());
if (k2<0) return -1;
int k3 = content.find(after,k2+before.length());
if (k3<0) return -1;
pos = k2 + before.length();
res = content.substr(pos,k3-pos);
return pos;
}
int CText::findbetween(string &res, const string &before,
const string &after, int pos = 0)
{
return findafter(res,"",before,after,pos);
}
int CText::geturl(string &res, const string &pattern)
{
int line = findline(pattern);
if (line<0) return -1;
int pos = findinline(res,line,"href=\"","\"");
return pos;
}
string CText::line(int index)
{
int start;
if (index==0) start = 0; else start = separators[index-1]+1;
int stop = separators[index];
return content.substr(start,stop-start);
}
vector<string> CText::harvest(const string &ldelimit, const string &udelimit,
const string &prefix, const string &before, const string &after)
{
return selectiveharvest("","",ldelimit,udelimit,prefix,before,after);
}
vector<string> CText::selectiveharvest(const string& starter,
const string& stopper, const string &ldelimit, const string &udelimit,
const string &prefix, const string &before, const string &after)
{
vector<string> res;
int beginning;
if (starter=="") beginning = 0; else beginning = content.find(starter);
if (beginning<0) return res;
int finish;
if (stopper=="") finish = content.length();
else finish = content.find(stopper,beginning);
if (finish<0) finish = content.length();
int start = content.find(ldelimit,beginning);
if (start<0) return res;
int stop = content.find(udelimit,start);
while ((stop>=0) && (start<finish))
{
string item;
int q = findafter(item,prefix,before,after,start);
if (q>stop) item="";
res.push_back(item);
start = content.find(ldelimit,stop);
if (start<0) return res;
stop = content.find(udelimit,start);
}
return res;
}
int CText::occurencies(const string& text)
{
if (text=="") return 0;
int count = 0;
int pos = 0;
int k = content.find(text);
while (k>=0)
{
count++;
pos = k+1;
k = content.find(text,pos);
}
return count;
}
vector<TDataRec> CText::parsetable(int pos)
{
vector<TDataRec> res;
string temp = content;
transform(temp.begin(),temp.end(),temp.begin(),toupper);
int p1 = temp.find("<TABLE",pos+1);
if (p1<0) return res;
int p2 = temp.find("</TABLE",p1+1);
if (p2<0) return res;
int p3 = temp.find("<TR",p1+1);
while ((p3>=0) && (p3<p2))
{
int p4 = temp.find("</TR",p3+1);
TDataRec row;
int p5 = temp.find("<T",p3+1);
while ((p5>=0) && (p5<p4))
{
int p6 = temp.find(">",p5+1)+1;
int p7 = temp.find("</",p6+1);
string cell = htmltostring(content.substr(p6,p7-p6));
row.push_back(cell);
p5 = temp.find("<T",p7+1);
}
res.push_back(row);
p3 = temp.find("<TR",p4+1);
}
return res;
}
Let me know your experience!
Saturday, July 11, 2015
My C++ utilities
When I started using C++, I was very disappointed by the lack of
many useful routines that were available to me in Delphi and PHP. So, I had to
write most of them myself. Over time, I accumulated my own library with the
most useful routines.
They are probably not super-well optimized but they do their
job. I don't create standalone applications that are distributed to end users.
I write programs to perform specific tasks on my computer – like analyzing data
or doing numerical analysis. For these purposes, my routines are doing quite
well.
Unfortunately, after switching from Microsoft Visual Studio
2010 to Visual Studio 2013, some of the API functions I was using turned out to be obsolete. I had to clean up what was too old (and
what was not very useful anymore). For example, I had a couple of nice date related routines but they all went down with the localtime function.
Here is a list of the functions (note that some examples require #include <iostream>):
Type conversion
int strtoint(const string &str)
Converts a string into an integer. Inspired by Delphi's strtoint. Example:
int val = strtoint("123");
string inttostr(int val, int length = 0)
Converts an integer into a string. If the resulting string is shorter than the length parameter, then function adds leading zeroes to make up for the difference. Inspired by Delphi's inttostr. Example:
string str = inttostr(8,2); // returns "08"
wchar_t* strtowchar(const string &lan)
Converts a string into a wchar_t*. Note that this requires manually deleting the string pointer later in the code in order to avoid memory leaks. Example:
wchar_t* fname = strtowchar("file.csv");
string wstrtostr(const wstring &lan)
Converts a wstring object into a string object. Example:
WIN32_FIND_DATA search_data;
[…]
filelist.push_back(wstrtostr(search_data.cFileName));
double strtodouble(const string &lan)
Converts a string into a real number. Inspired by Delphi's strtofloat. Example:
double num = strtodouble("23.7");
string doubletostr(double val)
Converts a real number into a string. Inspired by Delphi's floattostr. Example:
string str = doubletostr(17.4);
String operations
string trim(const string &lan)
Eliminates whitespace characters (space, tab, return, new line) at the beginning and at the end of a string. Inspired by PHP's trim. Example:
string str = trim(" test\r\n\t");
string compact(const string &lan)
Eliminates whitespace characters from the entire string. Example:
string str = compact(" t e s t ");
string str_replace(const string &source, const string &what, const string &dest, TReplacementType rt = RTAll)
Replaces occurrences of a substring in a string with something else. Depending on value of the rt parameters, it can replace all occurrences or only the first or the last occurrence (look up definition of the TReplacementType in my code for details). Inspired by PHP's str_replace. Example:
string str = str_replace("this is a test","is","at");
vector<string> explode(const string &source, const string &separator = "\n")
Converts a string into a vector of strings by breaking the given string at certain spots. The spots are indicated by the separator parameter. Inspired by PHP's explode. Example:
vector<string> vec = explode("Alice\nin\nWonderland");
string implode(const vector<string> &source, const string &separator = "\n")
Merges a vector of strings into a single string by joining the elements of the vector using a given separator string. This operation is opposite to explode. Inspired by PHP's implode. Example:
string str = implode(vec,"\n");
string htmltostring(const string &source)
Converts piece of HTML code to a text as it is (more or less) displayed by a browser. Example:
std::cout << htmltostring("<b>hey</b> \"<span class=\"q\">Jim</span>\"!");
string htmltocsv(const string &source)
Converts a piece of a HTML code into a string that is ready to be stored in a CSV file format which uses double quotes to delimit strings. Example:
std::cout << htmltocsv("<b>hey</b> \"<span class=\"q\">Jim</span>\"!");
bool isdigit(char c)
Returns true if the parameter is a digit or false otherwise. Example:
if (isdigit('7')) std::cout << "ok!";
int vecstrfind(const vector<string> &vec, const string &lan)
Returns an index of the vector of strings that points to a given string. Example:
int index = vecstrfind(vec,"Wonderland");
string up(const string& source)
Changes letter case of the string into the upper case. Example:
std::cout << up("Hello World!");
Handling files
bool fileexists(const string &filename)
Checks whether a file with a given file name exists. Example:
if (!fileexists("C:/Windows/System32/kernel32.dll")) std::cout << "oops!";
unsigned int getfilesize(const string &filename)
Returns the size of the file with a given file name. Example:
std::cout << getfilesize("c:/pagefile.sys");
bool download(const string &url, const string &filename)
Downloads a given URL and stores the response in a file. Example:
download("https://en.wikipedia.org/wiki/C%2B%2B","cpp.html");
vector<string> loaddirectory(const string &path)
Searches for files in a directory and returns their list in a vector of string. Wildcard characters are permitted. Example:
vector<string> filez = loaddirectory("c:/windows/system32/*.dll");
for (int i=0;i<filez.size();i++) std::cout << filez[i] << endl;
bool savetofile(const string &content, const string &filename)
Saves string to a file. Example:
savetofile("text,1\nme,2","file.csv");
vector<string> textfile(const string &filename)
Opens a file and reads its content into a vector of strings, each line of the file as a separate string. Inspired by PHP's file. Example:
vector<string> file = textfile("file.csv");
for (int i=0;i<file.size();i++) std::cout << file[i] << endl;
class CCSV
This class wraps a data object that allows for manipulations on a comma delimited file. The file may use (but does not need to) double quotes as text delimiters upon loading. The file will have double quotes as text delimiters upon saving. The data is stored in the data field which is a vector of vectors of strings. Example:
CCSV csv("file.csv");
csv.data[1][1] = "hello";
csv.savetofile("file.csv");
Random numbers
double random()
Generates a real random value uniformly distributed in [0,1). Inspired by PHP's rand. Example:
std::cout << random();
int dice(int minimum, int maximum)
Generates a random integer uniformly distributed between minimum and maximum, inclusive. Inspired by PHP's rand. Example:
std::cout << dice(1,6);
// The file's name is "anutil.h".
#pragma once
#include <sstream>
#include <fstream>
#include <UrlMon.h>
#include <vector>
#include <algorithm>
#include <iomanip>
#pragma comment(lib, "urlmon.lib")
using namespace std;
/* TYPE CONVERSION ROUTINES */
// Converts a string into an integer.
int strtoint(const string &str)
{
return atoi(str.c_str());
}
// Converts a string object into a wchar_t*.
// Note that this requires manually deleting the string
// pointer later in the code to avoid memory leaks.
wchar_t* strtowchar(const string &lan)
{
wchar_t *p = new wchar_t[lan.length()+1];
for (int i=0;i<lan.length();i++) p[i] = lan.at(i);
p[lan.length()] = '\0';
return p;
}
// Converts a wstring object into a string object.
string wstrtostr(const wstring &lan)
{
string s(lan.begin(), lan.end());
s.assign(lan.begin(), lan.end());
return s;
}
// Converts an integer value into a string object.
// If the resulting string is shorter than the "length" parameter,
// then function adds leading zeroes.
string inttostr(int val, int length = 0)
{
stringstream s;
s << val;
string res = s.str();
if (length>res.length())
{
string prefix(length-res.length(),'0');
return prefix+res;
} else return res;
}
// Converts a string object into a real number.
double strtodouble(const string &lan)
{
return atof(lan.c_str());
}
// Converts a real value into a string object.
string doubletostr(double val)
{
stringstream s;
s << setprecision(14) << val;
string res = s.str();
return res;
}
/* STRING PROCESSING ROUTINES */
// Eliminates whitespace characters (space, tab, return,
// new line) at the beginning and at the end of a string object.
string trim(const string &lan)
{
if (lan.length()==0) return lan;
int i = 0;
while ((lan[i]==' ') || (lan[i]=='\t') || (lan[i]=='\n') || (lan[i]=='\r'))
{
i++;
if (i==lan.length()) return "";
}
int j = 0;
while ((lan[lan.length()-1-j]==' ') || (lan[lan.length()-1-j]=='\t')
|| (lan[lan.length()-1-j]=='\n') || (lan[lan.length()-1-j]=='\r')) j++;
return lan.substr(i,lan.length()-i-j);
}
// Eliminates whitespace characters from a entire string.
string compact(const string &lan)
{
string res;
for (int i=0;i<lan.length();i++)
if ((lan[i]!=' ') && (lan[i]!='\t') && (lan[i]!='\n') && (lan[i]!='\r'))
res.push_back(lan[i]);
return res;
}
enum TReplacementType {RTAll, RTFirst, RTLast};
// Replaces occurrences of a substring in a string with something else.
// Depending on value of rt parameters it can replace all occurrences or only
// first or last occurrence.
string str_replace(const string &source, const string &what,
const string &dest, TReplacementType rt = RTAll) {
int index = 0;
string res = "";
do
{
int k;
if (rt==RTLast) k = source.rfind(what); else k = source.find(what,index);
if (k<0) return res + source.substr(index);
res += source.substr(index,k-index) + dest;
index = k + what.length();
if (index>=source.length()) return res;
} while (rt==RTAll);
return res + source.substr(index);
}
// Converts a string object into a vector of strings by breaking the given string
// at certain spots.The spots are given by the separator parameter.
vector<string> explode(const string &source, const string &separator = "\n") {
vector<string> res;
int index = 0;
int k = source.find(separator,index);
while (k>=0)
{
res.push_back(source.substr(index,k-index));
index = k+separator.length();
if (index<source.length()) k = source.find(separator,index); else k = -1;
}
if (index>=source.length()) res.push_back(""); else res.push_back(source.substr(index));
return res;
}
// Merges a vector of strings into a string by joining the elements of a vector
// with a given separator string. This operation is opposite to explode.
string implode(const vector<string> &source, const string &separator = "\n") {
stringstream s;
for (int i=0;i<source.size();i++)
{
if (i!=0) s << separator;
s << source[i];
}
return s.str();
}
// This routine converts piece of HTML code to a text as it is
// (more or less) displayed by a browser.
string htmltostring(const string &source)
{
string lan;
lan = str_replace(source,""","\"");
lan = str_replace(lan,"&","&");
lan = str_replace(lan,"'","'");
lan = str_replace(lan,"<br />","\n");
lan = str_replace(lan," "," ");
int lt = lan.find("<");
int gt = lan.find(">",lt);
while ((lt>=0) && (gt>=0))
{
string temp = lan.substr(0,lt);
if (gt+1<lan.size()) temp += " " + lan.substr(gt+1);
lan = temp;
lt = lan.find("<");
gt = lan.find(">",lt);
}
lan = str_replace(lan,"\n"," ");
lan = str_replace(lan,"\r"," ");
lan = str_replace(lan,"\t"," ");
string temp;
if (lan=="") return lan;
temp.push_back(lan[0]);
for (int i=1;i<lan.size();i++)
{
if ((lan[i]==' ') && (temp[temp.size()-1]==' ')) {} else temp.push_back(lan[i]);
}
return temp;
}
// This routine converts a piece of a HTML code into a string that is ready to be
// stored in a CSV file format which uses double quotes to delimit strings.
string htmltocsv(const string &source)
{
string lan;
lan = htmltostring(source);
lan = str_replace(lan,"\"","\"\"");
lan = "\"" + trim(lan) + "\"";
return lan;
}
// This routine returns true if the parameter is a digit or false otherwise.
bool isdigit(char c)
{
if (((int)c >= 48) && ((int)c <=57)) return true; else return false;
}
// This routine return an index of the vector of strings that points to a given string.
int vecstrfind(const vector<string> &vec, const string &lan)
{
for (int i=0; i<vec.size(); i++)
{
int w = lan.compare(vec[i]);
if (w==0) return i;
}
return -1;
}
// This routine changes letter case of the string into the upper case.
string up(const string& source)
{
string res = source;
transform(res.begin(),res.end(),res.begin(),::toupper);
return res;
}
/* FILE ROUTINES */
// Checks whether a file with a given filename exists.
bool fileexists(const string &filename)
{
wchar_t* fname = strtowchar(filename);
WIN32_FIND_DATA search_data;
memset(&search_data, 0, sizeof(WIN32_FIND_DATA));
HANDLE handle = FindFirstFile(fname, &search_data);
delete [] fname;
if (handle != INVALID_HANDLE_VALUE) return true; else return false;
}
// Returns the size of the file with a given file name.
unsigned int getfilesize(const string &filename)
{
wchar_t* fname = strtowchar(filename);
WIN32_FIND_DATA search_data;
memset(&search_data, 0, sizeof(WIN32_FIND_DATA));
HANDLE handle = FindFirstFile(fname, &search_data);
delete [] fname;
if (handle != INVALID_HANDLE_VALUE)
return (search_data.nFileSizeHigh * (MAXDWORD+1)) + search_data.nFileSizeLow;
else return -1;
}
// Downloads a given url and stores the response in a file.
bool download(const string &url, const string &filename)
{
wchar_t filename1[250];
for (int i=0;i<filename.length();i++) filename1[i] = filename.at(i);
filename1[filename.length()] = '\0';
wchar_t url1[250];
for (int i=0;i<url.length();i++) url1[i] = url.at(i);
url1[url.length()] = '\0';
return (URLDownloadToFile(NULL,url1,filename1,0,NULL)==S_OK);
}
// Looks up for files in a directory and returns their list in a vector of string.
// Wildcard characters are permitted!
vector<string> loaddirectory(const string &path)
{
wchar_t* kupa = strtowchar(path);
WIN32_FIND_DATA search_data;
vector<string> filelist;
memset(&search_data, 0, sizeof(WIN32_FIND_DATA));
HANDLE handle = FindFirstFile(kupa, &search_data);
delete [] kupa;
while(handle != INVALID_HANDLE_VALUE)
{
filelist.push_back(wstrtostr(search_data.cFileName));
if(FindNextFile(handle, &search_data) == FALSE) break;
}
return filelist;
}
// Saves string to a file.
bool savetofile(const string &content, const string &filename)
{
ofstream outfile;
outfile.open(filename);
if (!outfile.is_open()) return false;
outfile << content;
outfile.close();
return true;
}
// Opens a file and reads ints content into a vector of strings,
// each string is a separate line of the file.
vector<string> textfile(const string &filename)
{
ifstream plik(filename,ios::binary);
vector<string> res;
if (!plik.is_open()) return res;
string line;
while (!plik.eof())
{
getline(plik,line);
res.push_back(line);
}
return res;
}
typedef vector<string> TDataRec;
typedef vector<TDataRec> TTable;
// This class represents a data object that allows for manipulations on a comma
// delimited file with double quotes as text delimiters.
// The data is stored in the TTable object which is a vector of vectors of strings.
class CCSV {
public:
TTable data;
CCSV() {};
CCSV(const string& filename);
bool loadfromfile(const string& filename);
bool savetofile(const string& filename);
};
CCSV::CCSV(const string& filename)
{
loadfromfile(filename);
}
bool CCSV::loadfromfile(const string& filename)
{
ifstream infile(filename);
string line;
while (!infile.eof())
{
getline(infile,line);
if (compact(line)!="") {
int status = 0; // 0: outside a string; 1: inside a string
string lan = "";
TDataRec rec;
for (int i=0; i<line.size(); i++)
{
switch (status) {
case 0:
{
if (line[i]=='"')
{
status = 1;
break;
}
if ((line[i]!='"') && (line[i]!=','))
{
lan += line[i];
if (i+1==line.size())
{
rec.push_back(lan);
lan = "";
}
break;
}
if (line[i]==',')
{
rec.push_back(lan);
lan = "";
break;
}
break;
}
case 1:
{
if (line[i]=='"')
{
if ((i+1<line.size()) && (line[i+1]=='"'))
{
i++;
lan += '"';
} else {
i++;
rec.push_back(lan);
lan = "";
status = 0;
}
} else {
lan += line[i];
}
break;
}
}
}
data.push_back(rec);
}
}
infile.close();
return true;
}
bool CCSV::savetofile(const string& filename)
{
ofstream outfile(filename);
for (int i=0; i<data.size(); i++)
{
for (int j=0;j<data[i].size();j++)
{
string lan = "\"" + str_replace(data[i][j],"\"","\"\"") + "\"";
outfile << lan;
if (j+1==data[i].size()) outfile << endl; else outfile << ",";
}
}
outfile.close();
return true;
}
/* RANDOM NUMBER ROUTINES */
// Generates a random real value uniformly distributed in [0,1).
double random()
{
double randmax = (double)(RAND_MAX + 1);
double res = 0;
for (int i=0;i<4;i++)
{
res += (double)rand();
res = res / randmax;
}
return res;
}
// Generates a random integer uniformly distributed between minimum and maximum, inclusive.
int dice(int minimum, int maximum)
{
return random()*(maximum-minimum+1)+minimum;
}
#pragma once
#include <sstream>
#include <fstream>
#include <UrlMon.h>
#include <vector>
#include <algorithm>
#include <iomanip>
#pragma comment(lib, "urlmon.lib")
using namespace std;
/* TYPE CONVERSION ROUTINES */
// Converts a string into an integer.
int strtoint(const string &str)
{
return atoi(str.c_str());
}
// Converts a string object into a wchar_t*.
// Note that this requires manually deleting the string
// pointer later in the code to avoid memory leaks.
wchar_t* strtowchar(const string &lan)
{
wchar_t *p = new wchar_t[lan.length()+1];
for (int i=0;i<lan.length();i++) p[i] = lan.at(i);
p[lan.length()] = '\0';
return p;
}
// Converts a wstring object into a string object.
string wstrtostr(const wstring &lan)
{
string s(lan.begin(), lan.end());
s.assign(lan.begin(), lan.end());
return s;
}
// Converts an integer value into a string object.
// If the resulting string is shorter than the "length" parameter,
// then function adds leading zeroes.
string inttostr(int val, int length = 0)
{
stringstream s;
s << val;
string res = s.str();
if (length>res.length())
{
string prefix(length-res.length(),'0');
return prefix+res;
} else return res;
}
// Converts a string object into a real number.
double strtodouble(const string &lan)
{
return atof(lan.c_str());
}
// Converts a real value into a string object.
string doubletostr(double val)
{
stringstream s;
s << setprecision(14) << val;
string res = s.str();
return res;
}
/* STRING PROCESSING ROUTINES */
// Eliminates whitespace characters (space, tab, return,
// new line) at the beginning and at the end of a string object.
string trim(const string &lan)
{
if (lan.length()==0) return lan;
int i = 0;
while ((lan[i]==' ') || (lan[i]=='\t') || (lan[i]=='\n') || (lan[i]=='\r'))
{
i++;
if (i==lan.length()) return "";
}
int j = 0;
while ((lan[lan.length()-1-j]==' ') || (lan[lan.length()-1-j]=='\t')
|| (lan[lan.length()-1-j]=='\n') || (lan[lan.length()-1-j]=='\r')) j++;
return lan.substr(i,lan.length()-i-j);
}
// Eliminates whitespace characters from a entire string.
string compact(const string &lan)
{
string res;
for (int i=0;i<lan.length();i++)
if ((lan[i]!=' ') && (lan[i]!='\t') && (lan[i]!='\n') && (lan[i]!='\r'))
res.push_back(lan[i]);
return res;
}
enum TReplacementType {RTAll, RTFirst, RTLast};
// Replaces occurrences of a substring in a string with something else.
// Depending on value of rt parameters it can replace all occurrences or only
// first or last occurrence.
string str_replace(const string &source, const string &what,
const string &dest, TReplacementType rt = RTAll) {
int index = 0;
string res = "";
do
{
int k;
if (rt==RTLast) k = source.rfind(what); else k = source.find(what,index);
if (k<0) return res + source.substr(index);
res += source.substr(index,k-index) + dest;
index = k + what.length();
if (index>=source.length()) return res;
} while (rt==RTAll);
return res + source.substr(index);
}
// Converts a string object into a vector of strings by breaking the given string
// at certain spots.The spots are given by the separator parameter.
vector<string> explode(const string &source, const string &separator = "\n") {
vector<string> res;
int index = 0;
int k = source.find(separator,index);
while (k>=0)
{
res.push_back(source.substr(index,k-index));
index = k+separator.length();
if (index<source.length()) k = source.find(separator,index); else k = -1;
}
if (index>=source.length()) res.push_back(""); else res.push_back(source.substr(index));
return res;
}
// Merges a vector of strings into a string by joining the elements of a vector
// with a given separator string. This operation is opposite to explode.
string implode(const vector<string> &source, const string &separator = "\n") {
stringstream s;
for (int i=0;i<source.size();i++)
{
if (i!=0) s << separator;
s << source[i];
}
return s.str();
}
// This routine converts piece of HTML code to a text as it is
// (more or less) displayed by a browser.
string htmltostring(const string &source)
{
string lan;
lan = str_replace(source,""","\"");
lan = str_replace(lan,"&","&");
lan = str_replace(lan,"'","'");
lan = str_replace(lan,"<br />","\n");
lan = str_replace(lan," "," ");
int lt = lan.find("<");
int gt = lan.find(">",lt);
while ((lt>=0) && (gt>=0))
{
string temp = lan.substr(0,lt);
if (gt+1<lan.size()) temp += " " + lan.substr(gt+1);
lan = temp;
lt = lan.find("<");
gt = lan.find(">",lt);
}
lan = str_replace(lan,"\n"," ");
lan = str_replace(lan,"\r"," ");
lan = str_replace(lan,"\t"," ");
string temp;
if (lan=="") return lan;
temp.push_back(lan[0]);
for (int i=1;i<lan.size();i++)
{
if ((lan[i]==' ') && (temp[temp.size()-1]==' ')) {} else temp.push_back(lan[i]);
}
return temp;
}
// This routine converts a piece of a HTML code into a string that is ready to be
// stored in a CSV file format which uses double quotes to delimit strings.
string htmltocsv(const string &source)
{
string lan;
lan = htmltostring(source);
lan = str_replace(lan,"\"","\"\"");
lan = "\"" + trim(lan) + "\"";
return lan;
}
// This routine returns true if the parameter is a digit or false otherwise.
bool isdigit(char c)
{
if (((int)c >= 48) && ((int)c <=57)) return true; else return false;
}
// This routine return an index of the vector of strings that points to a given string.
int vecstrfind(const vector<string> &vec, const string &lan)
{
for (int i=0; i<vec.size(); i++)
{
int w = lan.compare(vec[i]);
if (w==0) return i;
}
return -1;
}
// This routine changes letter case of the string into the upper case.
string up(const string& source)
{
string res = source;
transform(res.begin(),res.end(),res.begin(),::toupper);
return res;
}
/* FILE ROUTINES */
// Checks whether a file with a given filename exists.
bool fileexists(const string &filename)
{
wchar_t* fname = strtowchar(filename);
WIN32_FIND_DATA search_data;
memset(&search_data, 0, sizeof(WIN32_FIND_DATA));
HANDLE handle = FindFirstFile(fname, &search_data);
delete [] fname;
if (handle != INVALID_HANDLE_VALUE) return true; else return false;
}
// Returns the size of the file with a given file name.
unsigned int getfilesize(const string &filename)
{
wchar_t* fname = strtowchar(filename);
WIN32_FIND_DATA search_data;
memset(&search_data, 0, sizeof(WIN32_FIND_DATA));
HANDLE handle = FindFirstFile(fname, &search_data);
delete [] fname;
if (handle != INVALID_HANDLE_VALUE)
return (search_data.nFileSizeHigh * (MAXDWORD+1)) + search_data.nFileSizeLow;
else return -1;
}
// Downloads a given url and stores the response in a file.
bool download(const string &url, const string &filename)
{
wchar_t filename1[250];
for (int i=0;i<filename.length();i++) filename1[i] = filename.at(i);
filename1[filename.length()] = '\0';
wchar_t url1[250];
for (int i=0;i<url.length();i++) url1[i] = url.at(i);
url1[url.length()] = '\0';
return (URLDownloadToFile(NULL,url1,filename1,0,NULL)==S_OK);
}
// Looks up for files in a directory and returns their list in a vector of string.
// Wildcard characters are permitted!
vector<string> loaddirectory(const string &path)
{
wchar_t* kupa = strtowchar(path);
WIN32_FIND_DATA search_data;
vector<string> filelist;
memset(&search_data, 0, sizeof(WIN32_FIND_DATA));
HANDLE handle = FindFirstFile(kupa, &search_data);
delete [] kupa;
while(handle != INVALID_HANDLE_VALUE)
{
filelist.push_back(wstrtostr(search_data.cFileName));
if(FindNextFile(handle, &search_data) == FALSE) break;
}
return filelist;
}
// Saves string to a file.
bool savetofile(const string &content, const string &filename)
{
ofstream outfile;
outfile.open(filename);
if (!outfile.is_open()) return false;
outfile << content;
outfile.close();
return true;
}
// Opens a file and reads ints content into a vector of strings,
// each string is a separate line of the file.
vector<string> textfile(const string &filename)
{
ifstream plik(filename,ios::binary);
vector<string> res;
if (!plik.is_open()) return res;
string line;
while (!plik.eof())
{
getline(plik,line);
res.push_back(line);
}
return res;
}
typedef vector<string> TDataRec;
typedef vector<TDataRec> TTable;
// This class represents a data object that allows for manipulations on a comma
// delimited file with double quotes as text delimiters.
// The data is stored in the TTable object which is a vector of vectors of strings.
class CCSV {
public:
TTable data;
CCSV() {};
CCSV(const string& filename);
bool loadfromfile(const string& filename);
bool savetofile(const string& filename);
};
CCSV::CCSV(const string& filename)
{
loadfromfile(filename);
}
bool CCSV::loadfromfile(const string& filename)
{
ifstream infile(filename);
string line;
while (!infile.eof())
{
getline(infile,line);
if (compact(line)!="") {
int status = 0; // 0: outside a string; 1: inside a string
string lan = "";
TDataRec rec;
for (int i=0; i<line.size(); i++)
{
switch (status) {
case 0:
{
if (line[i]=='"')
{
status = 1;
break;
}
if ((line[i]!='"') && (line[i]!=','))
{
lan += line[i];
if (i+1==line.size())
{
rec.push_back(lan);
lan = "";
}
break;
}
if (line[i]==',')
{
rec.push_back(lan);
lan = "";
break;
}
break;
}
case 1:
{
if (line[i]=='"')
{
if ((i+1<line.size()) && (line[i+1]=='"'))
{
i++;
lan += '"';
} else {
i++;
rec.push_back(lan);
lan = "";
status = 0;
}
} else {
lan += line[i];
}
break;
}
}
}
data.push_back(rec);
}
}
infile.close();
return true;
}
bool CCSV::savetofile(const string& filename)
{
ofstream outfile(filename);
for (int i=0; i<data.size(); i++)
{
for (int j=0;j<data[i].size();j++)
{
string lan = "\"" + str_replace(data[i][j],"\"","\"\"") + "\"";
outfile << lan;
if (j+1==data[i].size()) outfile << endl; else outfile << ",";
}
}
outfile.close();
return true;
}
/* RANDOM NUMBER ROUTINES */
// Generates a random real value uniformly distributed in [0,1).
double random()
{
double randmax = (double)(RAND_MAX + 1);
double res = 0;
for (int i=0;i<4;i++)
{
res += (double)rand();
res = res / randmax;
}
return res;
}
// Generates a random integer uniformly distributed between minimum and maximum, inclusive.
int dice(int minimum, int maximum)
{
return random()*(maximum-minimum+1)+minimum;
}
More goodies will come.
Subscribe to:
Posts (Atom)