Data scraping with C++

Hello. 

My name is CText. I am a C++ class. I can read a text file – usually an HTML or XML file. I can extract a lot of useful information from it. I can find a piece of text enclosed by given strings. I can look for it in a particular line or in the entire file. I can parse tables. I can parse lists. I have worked with various data sources, including Amazon, Christies, EPO, Factiva, and many more. I significantly speed up coding and increase code reliability. Although I am probably not perfectly optimized, I am doing my job well. I have to admit that I have been tested only with Visual C++ on Microsoft Windows. And I require an additional header file to function. But I still hope I can be useful. The following program illustrates my capabilities. The program

1. downloads the GDP data from the World Bank website,
2. converts the GDP data and saves it to a CSV file,
3. finds the first line of the table in the file and displays this line, and
4. retrieves the list of the countries and displays it.

Sample code

Download the source code or copy and paste it:

#include <iostream>

#include <andatathresher.h>

using namespace std;

int main()

{

cout << "Hello World!" << endl;

download("http://data.worldbank.org/indicator/NY.GDP.MKTP.CD", 

"file.html");

CText txt("file.html");

CCSV csv;

csv.data = txt.parsetable(0);

csv.savetofile("gdp.csv");

int line = txt.findline("<table");

cout << "The line containing first \"table\" has the number " 

<< line << " and its contents is: " << endl << txt.line(line) 

<< endl;

vector<string> ctrs = txt.selectiveharvest("Country name", 

"</table>", "<tr", "</tr>", "<a href", ">", "<");

cout << "Here is the list of countries covered:" << endl;

for (int i = 0; i < ctrs.size() - 1; i++) 

cout << ctrs[i] << "; ";

if (ctrs.size()>0) cout << ctrs[ctrs.size() - 1] << endl;

system("pause");

return 0;

}

Class members

string content

A field with the content of the file.

vector<int> separators

A field that contains positions of new line characters in the file.

CText::CText(const string &fname)

A class constructor. Loads a file and performs its preliminary analysis. A constructor without parameters can be called as well and file can be loaded manually but one has to be careful about functions using lines.  

bool CText::load(const string &filename)

This function loads a file with a given file name, stores its content in the field content, and performs its primary analysis. Returns true upon success and false otherwise.

void CText::update()

A method performing preliminary analysis of the file. It updates the separators vector.

int CText::findline(const string &text, int pos = 0)

This function returns the number of the first line that contains given string. It starts searching in the file from the offset pos. Returns -1 if no line is found.

int CText::findinline(string &res, int line, const string &before, const string &after)

This function extracts a string from a line with the line number given by line. The string that is directly after before and before after is returned through reference res. The function returns offset of the found string or -1 if it cannot be found.

int CText::findafter(string &res, const string &prefix, const string &before, const string &after, int pos = 0)

This function extracts a string from a file. It starts searching from the offset pos. It looks for the first subsequent occurrence of prefix and then for the first subsequent occurrence of before. The string that is directly after before and before after is returned through reference res. The function returns offset of the found string or -1 if it cannot be found.

int CText::findbetween(string &res, const string &before, const string &after, int pos = 0)

This function acts as findafter but with an empty string in prefix.

int CText::geturl(string &res, const string &pattern)

This function looks for a line that contains string pattern. Then it searcher for the first href HTML attribute it can find and returns its content trough reference res. The function returns offset of the found string or -1 if it cannot be found.

string CText::line(int index)

This function returns line from the file with the line number given by index. Lines are numbered from 0.

vector<string> CText::harvest(const string &ldelimit, const string &udelimit, const string &prefix, const string &before, const string &after)

This function acts as selectiveharves but with empty strings in starter and stopper. That is, it operates on the entire file.

vector<string> CText::selectiveharvest(const string& starter, const string& stopper, const string &ldelimit, const string &udelimit, const string &prefix, const string &before, const string &after)

This functions locates all strings that satisfy some criteria and returns them as a vector. It operates only on the fragment of the file that is after first occurrence of starter and before first subsequent occurrence of stopper. Within this range function looks for pieces of text enclosed by ldelimit and udelimit. Each such enclosure is supposed to produce one element of the resulting vector. Within each enclosure, function performs procedure similar to findafter and adds the result as an element to the resulting vector.

int CText::occurencies(const string& text)

This function counts how many times given string can be found in the file.

vector<TDataRec> CText::parsetable(int pos)

This function parses an HTML table and returns it as a vector of vectors of strings. It looks for the first table after offset pos. It will not work for tables inside tables. For the definition of TDataRec type see file anutil.h.

Class code

Download the source code or copy and paste it:

#pragma once

#include <string>

#include <vector>

#include "anutil.h"

#include <algorithm>

using namespace std;

/* INTERFACE */

class CText {

public:

string content;

vector<int> separators;

CText() {};

CText(const string &fname);

void update();

bool load(const string &filename);

int findline(const string &text, int pos);

int findinline(string &res, int line, 

const string &before, const string &after);

int findafter(string &res, const string &prefix, 

const string &before, const string &after, int pos);

int findbetween(string &res, const string &before, 

const string &after, int pos);

int geturl(string &res, const string &pattern);

vector<string> harvest(const string &ldelimit, 

const string &udelimit, const string &prefix, 

const string &before, const string &after);

vector<string> selectiveharvest(const string &starter, 

const string &stopper, const string &ldelimit, 

const string &udelimit, const string &prefix, 

const string &before, const string &after);

string line(int index);

int occurencies(const string &text);

vector<TDataRec> parsetable(int pos);

};

/* IMPLEMENTATION */

CText::CText(const string &fname)

{

load(fname);

}

void CText::update()

{

separators.clear();

for (int i=0;i<content.length();i++) 

if (content[i]=='\n') separators.push_back(i);

separators.push_back(content.length());

}

bool CText::load(const string &filename)

{

ifstream infile;

infile.open(filename,ios::binary);

if (!infile.is_open())

{

content = "";

return false;

}

stringstream cont;

string line;

while (!infile.eof())

{

getline(infile,line);

cont << line << endl;

}

content = cont.str();

infile.close();

update();

return true;

} 

int CText::findline(const string &text, int pos = 0)

{

if (separators.size()==0) return -1;

int k = content.find(text,pos);

if (k<0) return -1;

int i; 

for (i=separators.size()-1; i>=0; i--) 

if (separators[i]<k) break;

return i+1;

}

int CText::findinline(string &res, int line, 

const string &before, const string &after)

{

int start = content.find(before,separators[line]);

int stop = content.find(after,start+before.length());

if ((start<0) || (stop<0)) return -1;

if (line+1<separators.size()) if ((start>separators[line+1]) 

|| (stop>separators[line+1])) return -1;

int pos = start + before.length();

res = content.substr(pos,stop-pos);

return pos;

}

int CText::findafter(string &res, const string &prefix, 

const string &before, const string &after, int pos = 0)

{

int k1 = content.find(prefix,pos);

if (k1<0) return -1;

int k2 = content.find(before,k1+prefix.length());

if (k2<0) return -1;

int k3 = content.find(after,k2+before.length());

if (k3<0) return -1;

pos = k2 + before.length();

res = content.substr(pos,k3-pos);

return pos;

}

int CText::findbetween(string &res, const string &before, 

const string &after, int pos = 0)

{

return findafter(res,"",before,after,pos);

}

int CText::geturl(string &res, const string &pattern)

{

int line = findline(pattern);

if (line<0) return -1;

int pos = findinline(res,line,"href=\"","\"");

return pos;

}

string CText::line(int index)

{

int start;

if (index==0) start = 0; else start = separators[index-1]+1;

int stop = separators[index];

return content.substr(start,stop-start);

}

vector<string> CText::harvest(const string &ldelimit, const string &udelimit, 

const string &prefix, const string &before, const string &after)

{

return selectiveharvest("","",ldelimit,udelimit,prefix,before,after);

}

vector<string> CText::selectiveharvest(const string& starter, 

const string& stopper, const string &ldelimit, const string &udelimit, 

const string &prefix, const string &before, const string &after)

{

vector<string> res;

int beginning;

if (starter=="") beginning = 0; else beginning = content.find(starter);

if (beginning<0) return res;

int finish;

if (stopper=="") finish = content.length(); 

else finish = content.find(stopper,beginning);

if (finish<0) finish = content.length();

int start = content.find(ldelimit,beginning);

if (start<0) return res;

int stop = content.find(udelimit,start);

while ((stop>=0) && (start<finish))

{

string item;

int q = findafter(item,prefix,before,after,start);

if (q>stop) item="";

res.push_back(item);

start = content.find(ldelimit,stop);

if (start<0) return res;

stop = content.find(udelimit,start);

}

return res;

}

int CText::occurencies(const string& text)

{

if (text=="") return 0;

int count = 0;

int pos = 0;

int k = content.find(text);

while (k>=0)

{

count++;

pos = k+1;

k = content.find(text,pos);

}

return count;

}

vector<TDataRec> CText::parsetable(int pos)

{

vector<TDataRec> res;

string temp = content;

transform(temp.begin(),temp.end(),temp.begin(),toupper);

int p1 = temp.find("<TABLE",pos+1);

if (p1<0) return res;

int p2 = temp.find("</TABLE",p1+1);

if (p2<0) return res;

int p3 = temp.find("<TR",p1+1);

while ((p3>=0) && (p3<p2))

{

int p4 = temp.find("</TR",p3+1);

TDataRec row;

int p5 = temp.find("<T",p3+1);

while ((p5>=0) && (p5<p4))

{

int p6 = temp.find(">",p5+1)+1;

int p7 = temp.find("</",p6+1);

string cell = htmltostring(content.substr(p6,p7-p6));

row.push_back(cell);

p5 = temp.find("<T",p7+1);

}

res.push_back(row);

p3 = temp.find("<TR",p4+1);

}

return res;

}

Let me know your experience!

Alien invasion

‘Yes, my little fish. What is your worry?’ said papa fish folding newspaper.

‘Is there life outside the Pond?’ asked little fish.

‘There probably is. The Universe is very big... And it seems to be made out of things we are made of. So extra-pondial life is very likely to exist somewhere’ explained papa fish. ‘In fact’ he continued ‘we know that there are other ponds out there similar to ours. We have already found some.’

‘Has anybody gone there?’ asked little fish.

‘We don't have the technology yet. We have some flying fish which can jump out of water for a few seconds, but to go to a different pond… it's much more complicated.’

‘So how could life look like in these different ponds?’ little fish kept inquisitive.

‘Oh, there is actually something that we can expect. There are features of animals here in the Pond that evolved separately multiple times and seem to be quite useful. We can expect that things like gills and fins are likely to evolve in a different pond as well. Of course, it is silly to depict aleins as fish-morphic creatures as they often are in the movies. Look for example how different octopi are from us. But they have gills too.’ 

‘Hm…’ little fish was thinking for a while. ‘When we meet them, will these aliens be friendly or hostile? Will they try to conquer the Pond?’

‘We can't be sure. But I believe they will be friendly. We can exchange technology. And there is a plenty of uninhabited ponds. Why would they need ours?’

After the conversation, little fish went to bed. Unfortunately, it was the last conversation he had with papa fish. Little they knew that as they spoke a band of humans rolled out their camping site next to the Pond. Next day, early in the morning, one of the humans cast a net and caught both little fish and papa fish, as well as a few others. It took them 20 minutes to die a painful death of suffocation in a fisherman’s bucket. And then they were grilled and eaten.

It is interesting to imagine how  extraterrestrial life would look like. Finding even simple life would have great philosophical and scientific implications. But finding extraterrestrial intelligence is especially interesting. We could learn a lot about ourselves just by observing other intelligence. Many things we take as general truths could turn out to be products of our minds. Also, if alien intelligence shows up somewhere close by, we may end up competing with them for resources. And if they are technologically superior, then we are screwed. So let us see what top scientific minds can say about how aliens would look like and how our encounter with them could unfold:

• Into the Universe with Stephen Hawking: Aliens
• The Poetry of Science: Richard Dawkins and Neil deGrasse Tyson
• Curiosity - Alien Invasion: Are We Ready?
• Naked Science - Alien Contact
• The Fermi Paradox II — Solutions and Ideas – Where Are All The Aliens?

I am still very disappointed by anthropomorphism and "biomorphism" of the creatures they are discussing (maybe except for the last one). I want to strip these visions from their biases and make them as realistic as possible. Although, I may identify some shortcomings but my thinking will be still preconditioned by the experience of Earthly life. Therefore, all I hope to achieve is to make an incremental improvement and pass the work on to somebody who will identify the biases I overlooked. We deserve better aliens!

Meet a planet-eater

I present you a planet-eater. This creature is a sphere, similar in size of the Imperial Death Star (160 kilometers/100 miles in diameter) and weights around 5 thousand trillion tons. It has a thick shell that along with self-generated magnetic and electric fields protect the interior from cosmic radiation and other dangers. The shell is able to withstand a high-yield nuclear blast, as it is at least 20 kilometers (12 miles) thick. The bowels of the creature contain among others resource silos as well as mechanical and chemical plants that allow the planet-eater to create any device it needs. It is also equipped with a fission and fusion cores that allow morphing some elements into others.

The propulsion systems are built as they are needed. They attach themselves to the creature’s surface and operate as long as necessary. Then, they are hauled back inside where they are disassembled. The internal factories are also able to manufacture probes, communication devices, smaller crafts able to capture and haul asteroids or comets, repair robots, defense drones, exterior silos, and so on. The list of blueprints includes everything that is needed to be self-sufficient in the interstellar space. More devices can be designed on the spot as needed.

The creature can equip itself with a multitude of sensors. Data from these sensors are analyzed by an on-board supercomputer whose intelligence is orders of magnitude higher than the total intelligence of all humans who ever lived combined. The volume of the central unit is around two cubic kilometers (0.8 cubic miles) and weights 4 billion tons. The supercomputer has access to sensors inside the creature’s body. It also monitors itself, thus meeting all necessary conditions for being self-conscious.

Planet-eaters are semi-social species. It is hard to make a random encounter in the interstellar space and if one happens, the help usually arrives no sooner that after a few years. Each planet-eater is thus equipped to do well on its own. However, whenever two planet-eaters meet, they quickly exchange information they accumulated about the Universe and work together to see if they can improve their blueprints or even totally redesign themselves. They can work together to fend off enemies or solve problems one planet-eater could not deal with on its own. There have been even cases of self-sacrifice. Their social interactions can get very complicated and are intractable for a simple human brain.

Planet-eaters are very rational when it comes to interaction with other species. As with everything else, the decision on how to interact is based on cold calculation of costs and benefits. Unfortunately, less advanced creatures usually have nothing to offer them. They are at best ignored. Higher creatures on the other hand, would be valuable partners but they in turn are not interested in partnerships. Some of them just ignore planet-eaters; others use them as their food.

In fact, planet-eaters are one of the most primitive inhabitants of the Universe. They feed on primordial matter so they will run their course as soon as they eat up everything. They are small, week, and stupid as compared to other aliens, like algae are small, week, and stupid as compared to a whale. Finally, their ecological niche is small. The planets, asteroids, and dust clouds on which they can feed constitute just 1% of the Universe matter. The rest is contained in stars and other supermassive objects.

They are certainly dwarfed by their more advanced cousins: star-eaters (you may not realize that dark matter consists mostly of stars currently being consumed by star-eaters). But their evolutionary advantage is that they can multiply quickly, and they move relatively fast, so they can reach distant places before other life forms. In fact, they are already present in the Milky Way, and a number of planet-eaters are currently headed towards the Earth (as well as several other adjacent solar systems). The first three planet-eaters traveling together should enter our solar system within 15 thousand year. Others will come soon after.

Upon their arrival, planet-eaters will send probes to analyze contents of our solar system and to plan how to use its matter most efficiently. Running back and forth through the Solar System is not an easy due to the Sun’s gravity. The routs must be carefully planned to minimize energy expenditure.

As they approach our solar system, the planet-eaters will soon realize that there is something unusual going on in it. Namely, they will detect human activity. They will send probes to examine us. Initially, the probes will be orbiting from afar objects like Venus, the Earth, the Moon, Mars, Ceres, IO, Callisto, and Titan, that is all places with our significant presence.

They will intercept a lot of communication between us. These pieces of intelligence will allow the invaders, still undetected by us, to land some probes on the Earth and other celestial bodies colonized by us. Maybe a few abductions will take place (but bodies will be discarded rather than returned). Learning how we communicate will not be hard, and after connecting to the Internet, they will know about us everything they need to know. Obviously, they will learn nothing from us in terms of science. The most important thing to learn will be that we will defend the rocks we inhabit using all measures available to us, including nuclear weapons. We will be like insects crawling over a fruit and stinging anybody who wants to pick it up. No hard feelings, but you don’t want to be stung while eating, right?

While still in the outskirts of our solar system, the planet-eaters will start to feed on rocks from the Kuiper belt in order to prepare the invasion. After three dozen years of such preparations, the attacks will occur simultaneously in all major population centers. Every celestial body will have a form of attack most efficient for its circumstances. For example, the Earth will have its atmosphere poisoned in such a way that most biological life will go extinct within hours. Some facilities on the Moon will be nuked. Titan will have its atmosphere heated so much that all human installations there will get vaporized. And so on. There will be no need to engage in inefficient direct combat. Why would you sting insects back if you can just blow them off?

In a few hours, human civilization will be reduced to rubble. Only people in most remote space stations will survive. They will live not because they will have been overlooked or because planet-eaters have mercy. They will live because they pose no obstacle to further exploitation of the Solar System and there is no point in wasting energy on killing them.

No sooner than after several years the survivors will start to partially understand what happened to their civilization and who the adversaries are. But with limited spare parts and destroyed manufacturing capabilities, the last humans will die three centuries after initial attack. And the only reason to be proud of ourselves will be these three dozens years it took to prepare the alien invasion. If we were less advanced, say at the ninetieth century level, they would simply ignore our existence and accidentally vaporize us here and there and then leave the rest to freeze and suffocate. But the result will be similar. Even with all this advanced technology, the last future human will freeze to death in a leaking undersurface base on Iris.

By then, the planet-eaters will be busy consuming the planets of our solar system. There is no need to hurry – planet-eaters are usually safe during the feast. It is unlikely that a threatening object would suddenly appear in their vicinity, because flying even from the closest star would take at least four years. And communication with neighboring systems will be indicating that the area is safe.

A contingent of harvesters manufactured in the belly of one of the planet-eaters will land on the surface of the Earth. The machines will gather resources and fly back to their master to replenish contents of its resource silos. And while the planet-eater on the Earth’s orbit will be busy manufacturing new minions, the robots on the Earth will start the slow process of transforming Earth’s matter into bodies of new planet-eaters. It will take long, as it will require cooling down the Earth’s core (all this iron and uranium are especially valuable). But in the end, over hundred thousand copies will be produced and only debris floating on the former Earth’s orbit will be left. In total, the materials in the Solar System will be enough to create over a million of fully functional and independent copies of planet-eaters. The entire process will take around twenty thousand years. Then, the last planet-eater will fly away, leaving the lone Sun behind. Star-eaters will come twelve millions years later.

A case for planet-eaters

Are we likely to encounter creatures like intelligent humanoids or even spider-like aliens with silicon-based biology? We tend to create our aliens in our own image. And we tend to create invasion scenarios in a way we would probably invade. But we are in a short transitory state of technological and intellectual progress. We are unlikely to stay this way for long and it is unlikely that we will meet aliens resembling us, as we are right now. Aliens may be billions of years of evolution ahead. They will have been flying in space for billions of years. I expect a creature flying in space for that long to actually evolve capability to fly in space. Analogically, you would not expect terrestrial animals to carry around a bubble of water so that they can breathe with their gills.

But creation of first planet-eaters may be much quicker than billions of years. The idea of self-replicating robots is not hard to come by in science fiction. Add ability to travel in space and superintelligence and you get a proto-planet-eater. It is likely that some earthling engineer will eventually construct such a thing... that is, if we survive long enough. 

The video by Kurz Gesagt is so good that I will address is separately. Many of their ideas resonate with me. However, self-replicating nanobots do not seem very likely to fare well on their own. They must be organized into a higher being. Otherwise, superintelligence is out of question. Without intelligence, they will lack adaptability and could be easily defeated by something with intelligence. Also, populating virtual world in a cozy neighborhood of a red dwarf may be fun until a technologically superior star-eater comes along. Reality check. 

The key to thinking about aliens are evolution and natural selection. The species that survives is the species that is best at acquiring resources and making copies of themselves (or growing). Being intelligent and having advanced civilization is a tool to improve this feature rather than impede it. People sometimes feel noble as they try to protect the environment and other species from extinction. Then, they think that the progress of our civilization will eventually lead to the ultimate intra-special altruism and they project this notion on their imaginary advanced alien species. This is laughable. Don’t call yourself noble after giving to others some not-so-useful scraps. Call yourself noble only after you restrict your population growth so that there are more resources for other species. And even then, do not expect intelligent aliens to do the same. Such an act of altruism is an evolutionary disadvantage if you are competing with a different species for resources. On the Earth we no longer compete, so we indulge in self-apotheosis.

We humans tend to keep our heads in the heavens. But if we ever meet intelligent aliens, we will be brought down to earth rather quickly.

Will privatization save social security?

In this article I explain two arguments that I have never seen on the media or in any other setting, but which seem to be very important to the debate about privatization of social security. I do not want to give a general picture with an overview of pros and cons leading to a clear verdict. I just want to present these two apparently neglected arguments as my contribution to the debate. Also, I assume that you have some basic knowledge about how social security works. If you do not – please read my previousarticle about this topic. I am going to explain the arguments in detail first and at the end I will present a "cocktail party version." Sit tight.

Social Security is going bankrupt

Many people are concerned that social security is going bust due to excessive future expenditure and insufficient future revenues. This is indeed a legitimate concern. There will be problems if we continue business as usual, although the problems are not as serious as some people paint them. Nevertheless, the proponents of privatization use the prospect of bankruptcy as an argument in their favor. Current system, so called pay-as-you-go, depends on the contributions of current workers to fund pensions of current retirees. When contributions dry up and the number of workers per pensioner dwindles, the system is bound to collapse.

On the other hand, in the private system, workers would accumulate savings in their own private accounts. Upon retirement, pensions would be drawn from these accounts. Shortage of money would not occur and demographic change would not pose any threat. Therefore, using private sector instead of public sector to handle pensions would not have led to the problems we are facing today in the first place, and should avert similar problems in future.

This argument seems neat at the first glance. But it suffers from one serious flaw: it treats money as if its value does not change over time. In other words, this argument is a product money illusion.

By definition, economists are interested in allocation of resources in the economy. "Who will produce what goods?" and "who will consume what goods?" are the core questions of economics. Economists are interested in money, because money is a tool to allocate resources. But money in itself does not matter. The salience of money is derived from its ability to be exchanged for actual goods and services. When money loses this function (for example because of hyperinflation), it becomes irrelevant. Your material wellbeing does not depend of how much money you have but how much actual goods you can buy. Analyzing problem in terms of money is easy and often useful but sometimes it blurs the underlying nature of the problem. So let us forget about money for a while and analyze the problem in terms of goods and services.

There are two groups of people involved in the problem. Domestic workers create all goods and services that are available for sale in the economy (with the exception of imported goods, but I will neglect the international trade for clarity of reasoning). These goods are consumed by the workers, pensioners, and other groups (like children, unemployed, etc. – for simplicity, let us forget about them too). So how exactly are these goods allocated between workers and pensioners?

In ancient times each person was creating goods mostly for their own consumption. But modern economy benefits from division of labor, where efficiency of production and quality of services are increased thanks to specialization. Currently, each worker generates some amount of particular goods or services. Then, workers can exchange and share the fruits of their labor with other people. However, workers cannot take the entire value of what they create for themselves. They have to give a share to the company owners in exchange for capital and internal services like HR or IT (this is done automatically so there is no conscious act of "giving"). They also have to give a share to the government in form of taxes. Finally, they have to give a share to the elderly through social security.

To understand this process better, let us consider a very simplified version of the economy in which there are only eleven inhabitants of whom two are retired and nine are working. The economy is so simple, that the only goods are burgers. Each worker is able to create 1000 burgers per year. The people of this Burgerland decided to implement social security. There is a payroll tax of 10% which is paid to fund pensions. Payroll tax means that workers have to give 10% of the burgers they produce to the Social Security Administration (SSA). As a result, each worker consumes only 900 burgers per year.

Because there are nine workers, SSA receives 900 burgers a year. This is split between the two pensioners of whom each receives 450 burgers. As a result, the retirement replacement ratio, that is the value of pensions in relation to salary received before the retirement is 450/900 = 1/2. Every worker can expect to receive upon retirement half of what they were earning while working. In other words, thanks the payroll tax, the resources of the economy are redistributed to the elderly in such a way so that they have half as much per person as the workers.

Now, let us fast forward several decades. We observe that productivity increased thanks to technological progress.  Every worker is now able to create 1500 burgers, an increase in productivity of 50%. We also see that a demographic change has occurred. Although there are still eleven people in the country, only eight are workers and three are retirees.

The payroll tax has been kept at the same rate: 10%. Each worker contributes 150 burgers and consumes 1350 burgers. The total income of SSA is 8*150=1200 burgers and this is split equally among the three pensioners who receive 400 burgers each. You can see that the replacement ratio is 400/1350 which is less than 30%. Every worker expects to receive less than 30% of their salary upon retirement.  Compare it to the 50% in the previous situation and you see a significant drop in benefits!

This is similar to the situation which our real-life social security is bound to face in the future. There will be fewer people providing goods and services who pay payroll taxes and relatively more people buying these goods and services. As a result, the replacement ratio must fall or contributes must rise. These changes are imminent and will occur regardless of increases in productivity and changes to the country's GDP.

How is privatization supposed to solve this problem? Let us give privatization the benefit of the doubt. Let us say that in the first period the nine workers were saving 10% of their income – that is they were consuming 900 burgers a year each. The two retirees had 50% replacement rate – they were eating 450 burgers a year each. Now, fast forward several decades and we observe that economic progress increased productivity to 1500 burgers per worker per year. Thanks to privatization we managed to somehow keep the contributions at 10% and the replacement rate at 50%.

Everything is perfect... but wait a minute! There are eight workers, each consuming 1350 burgers a year. There are three retirees each consuming 675 burgers a year. So in total, there are 8*1350+3*675=12825 burgers consumed in this economy per year. But there are only 8*1500=12000 burgers produced in the economy per year! The numbers do not add up. In fact, there are not enough burgers for retirees to buy with the money they have. Something must have gone wrong!

This is the key to the problem. The root of the issue is the demographic change. There will be relatively fewer people creating goods and services and more retirees consuming these goods and services. Fewer producers and more consumers mean that somebody has to lose. Workers must share more with retirees or retirees must consume relatively less. There is no way around it. Privatizing social security is similar to using creative accounting to solve financial difficulties of a company. It does not make the problem go away, it just hides it temporarily. Privatization blurs the picture and makes us unable to identify who is going to be hit the most by the demographic change, whether they will be hit abruptly or gradually and when they are going to be hit. Note that factors like inflation, economic growth, and so on, are here irrelevant.

You may wonder how would a real life privatization scenario look like, as our Burgerland does not seem to be very realistic. If privatization was successful, we would have an increasing population of elderly with big savings account and a decreasing fraction of workers. They are areas of the country where elderly people tend to concentrate (e.g. Florida). In such places the number of people requesting services will increase and the number of people who can serve them will decrease most significantly. In order to compensate for the changes in demand for and supply of goods and services, prices will have to increase. The most affected will the de goods that the elderly buy.

Let us consider health care. The increasing number of elderly will force hospitals to employ more doctors. Either doctors will have to be paid more to entice people who would otherwise choose other professions, or less qualified people will become doctors out of necessity. Most likely, we can expect a combination of both. This results in higher prices and lower quality of health care. On top of that, since more doctors means fewer people available to other professions, the prices of other goods will rise as well, although not as much. Such inflation can potentially wipe out savings of the elderly clustered in places like Florida. And their life will be in general tougher. Imagine a town full of elderly in which the last grocery store is closed due to a lack of work force. This may be a stretch but it illustrates the type of problems they would face thanks to demographic change. Privatization of social security is not going to change it.

In a summary, the root of all evil is the demographic change. There will be relatively fewer people producing goods and more people consuming them. Somebody will have to lose. Thinking that privatizing social security will solve this problem is based on a mistake. Ensuring that money adds up is not enough. Value of money can change over time and what in the end needs to add up is not money but the amount of goods and services produced and consumed.

Privatization will lead to higher returns

The rate of return you receive from social security depends on your income. Social security was designed to reduce poverty among elderly. No wonder that people who made little money through their lives receive relatively higher pensions. For them, the rate of return is the highest and can reach above6 percent. For people making a lot of money, return on social security is negative – that is they pay more in payroll taxes than they receive in benefits.

Although the richest people would potentially benefit the most from the privatization of social security, the argument goes that it would boost efficiency of the system as a whole. The average rate of return from social security is currently quite low in comparison to average returns from, say, US stock market. If the money from payroll taxes was invested in the stock market, retirees would be surely better off.

Again, this argument makes sense at the first glance but falls apart after closer examination. The flaw here is the assumption that the same rate of return can be maintained for all these new savings created by the privatization. It is against the law of supply and demand. The more capital is there, the lower the interest rate.

Notice that in practice the money held in private retirement accounts would be taken care of by pension fund managers. To make a return on the contributions, they would have to buy some financial assets, be it stocks, bonds, or other securities. The higher demand for the financial assets, the higher their price and the lower the returns per amount invested.

Some people may think: yes, but availability of all these new capital will surely increase investment and thus lead to higher economic growth rates. That is to an extent true, but a clarification is needed. New enterprises and economic growth are not going to massively pop up out of nowhere just because new money for investment is available. New business ideas will probably continue to emerge at the same rate, just more of them will get funding. And guess what – financial markets tend to allocate capital to best, most promising, or safest investments. The new projects that would potentially get the additional money will be more risky and with lower expected profitability. In other words, increase in the amount of capital available will not magically increase the amount of capital needed. The shift of the capital supply curve to the right will not magically induce the shift of capital demand curve to the right. Rather, we will move along the demand for capital curve, towards the interest rate of zero.

Source: instructure.com

Of course, if the amount of new capital on financial markets is not too high, the change in the interest rate will be negligible. So what is the amount of savings that would be accumulated if social security was privatized? There are a few ways to estimate that amount. For simplicity, let us assume that people would contribute as much as they do today. We can then just take the total annual contributions to social security ($650 billion) and multiply it by the average number of years worked by a worker (40). As a result we get around $25 trillion dollars of savings that would be managed by the pension funds under privatized social security. This is more than capitalization of the entire US stock market. Such an amount will have a profound impact on rates of return.

How sensitive is the market interest rate to an increase in the supply of capital? As often in economics, it is probably impossible to discern a general economic law in this regard. The answer will most likely depend on numerous circumstances like the source of the funds, the risk preferences of the fund managers, legal constraints, current market conditions (are investors bullish or bearish?) and so on. Recent experiments with quantitative easing can give us a clue.

The objective of quantitative easing was to reduce long-term interest rates in order to spur economic activity. Central bank (Fed) buys very safe financial assets (e.g. government bonds), hoping that the investors who sold them would do something productive with their money, for example buy corporate bonds or make loans to companies. As recent research shows, there are many investors who have a preference for some class of financial assets. For example, some financial institutions are forced by law to put some fraction of their money in very safe assets like government bonds. They cannot simply substitute this investment with some more risky assets. As a result, quantitative easing has much weaker effect on the interest rate paid by debtors who contribute to economic growth (start-up companies, credit card owners, etc.) and a strong effect on interest rates of the assets that are actually purchased by the central bank (that is government bonds).

The amount of money spent during quantitative easing is an order of magnitude smaller than the market capitalization of the bonds that were purchased through it. Yet it had a significant effect on their interest rates. And now, we are to believe that dumping a pile of cash greater than the entire US stock market capitalization onto financial markets is not going to depress rates of return. No, the correct assumption is that the privatization of social security will depress interest rates significantly. If proponents of social security privatization argue otherwise, they must explain why this is not going to happen.

Let us summarize. Privatization of social security will generate a huge pile of savings. We know from experience that much smaller injections of capital into financial markets tend to depress interest rates. Assuming that the interest rate would be the same with and without privatization of social security is clearly nonsense.

Cocktail party version

If you want to show off at a cocktail party, you will need a simplified and concise version of the arguments. I do not guarantee, that everybody is going to understand them. Actually, I guarantee, that somebody is not going to understand them. But you can always try. Disclaimer: I do not take responsibility for tears and bruises.

1. Privatization of social security won’t save the underlying demographic problem. If you want to see why, imagine that no new people are born and eventually everybody in the country becomes retired. With privatized social security, it is very nice that everybody has their own accounts full of money. But there is nothing to spend this money on because everybody is retired and nobody is manufacturing goods and providing services! With social security as it is right now, the demographic problem can be clearly seen. With privatized social security we just do creative account to temporarily hide the problem. But we do not solve it!
2. Comparing historical stock market rates of return to historical social security rates of return does not make sense. Privatized social security will significantly increase the stock of savings in the economy. And we know that more savings means lower interest rates from the simple law of demand and supply.

As I mentioned, there are more economic arguments in this debate. In this article, I decided to focus on these two only.

Good luck with the rest.

The shape of the rainbow

Have you ever wondered how would the rainbow look like as seen by an animal that has different vision spectrum than human? Just posing this simple question should lead to a quick conclusion that we see just a part of the rainbow – the part that shines with the colors we can perceive. The rainbow extends both into infrared and into ultraviolet and the only reason we can’t see it, is because our eyes can’t perceive these “colors.”

If you were able to see all parts of the electromagnetic spectrum, you would see that the rainbow is made of a number of separated arcs of different width and brightness. This is because the Earth’s atmosphere is opaque to many light frequencies, for example high energy ultraviolet and most of the infrared spectrum. Also, keep in mind that the reason why rainbow emerges in the first place is because light rays going through water are scattered. Anything that cannot be scattered by water droplets is then out of the picture too. This includes some microwaves (water is opaque to microwaves, this is why microwaves boil water, duh!) and radio waves (anything with wavelength comparable to or exceeding the size of a droplet cannot be scattered by it).

Source: Wikipedia

As a matter of fact, if the lens in your eye was not opaque to ultraviolet light, you would be able to see it (and some people do, after removal of their lenses)! An interesting fact is that ultraviolet is not perceived by cones (which are the cells responsible for perceiving color) but lower wavelength ultraviolet is perceived by rods (which are the cells which perceive light intensity). Thus, if not for the lens, something glowing in ultraviolet would just look brighter, but would not give it any distinct color you could name. Your brain would not be able to distinguish between something glowing green and ultraviolet and something brighter glowing just green (or glowing green and white). If you looked at the rainbow, the arc next to violet would just make the colors of the background brighter.

Next time look carefully, maybe it does!

My C++ utilities

When I started using C++, I was very disappointed by the lack of many useful routines that were available to me in Delphi and PHP. So, I had to write most of them myself. Over time, I accumulated my own library with the most useful routines.

They are probably not super-well optimized but they do their job. I don't create standalone applications that are distributed to end users. I write programs to perform specific tasks on my computer – like analyzing data or doing numerical analysis. For these purposes, my routines are doing quite well.

Unfortunately, after switching from Microsoft Visual Studio 2010 to Visual Studio 2013, some of the API functions I was using turned out to be obsolete. I had to clean up what was too old (and what was not very useful anymore). For example, I had a couple of nice date related routines but they all went down with the localtime function.

An important caveat: I am using Visual C++ on Windows so the following code should be easy to use for people who use this platform. Just create anutil.h file containing my code, put the file in the same folder as your source code, and add the #include "anutil.h" line in the beginning of your code. People using Linux will have to pick my code apart, as it is platform dependent - it includes numerous references to Windows API.

Here is a list of the functions (note that some examples require #include <iostream>):

Type conversion

int strtoint(const string &str)
Converts a string into an integer. Inspired by Delphi's strtoint. Example:
int val = strtoint("123");

string inttostr(int val, int length = 0)
Converts an integer into a string. If the resulting string is shorter than the length parameter, then function adds leading zeroes to make up for the difference. Inspired by Delphi's inttostr. Example:
string str = inttostr(8,2); // returns "08"

wchar_t* strtowchar(const string &lan)
Converts a string into a wchar_t*. Note that this requires manually deleting the string pointer later in the code in order to avoid memory leaks. Example:
wchar_t* fname = strtowchar("file.csv");

string wstrtostr(const wstring &lan)
Converts a wstring object into a string object. Example:
WIN32_FIND_DATA search_data;
[…]
filelist.push_back(wstrtostr(search_data.cFileName));

double strtodouble(const string &lan)
Converts a string into a real number. Inspired by Delphi's strtofloat. Example:
double num = strtodouble("23.7");

string doubletostr(double val)
Converts a real number into a string. Inspired by Delphi's floattostr. Example:
string str = doubletostr(17.4); 

String operations

string trim(const string &lan)
Eliminates whitespace characters (space, tab, return, new line) at the beginning and at the end of a string. Inspired by PHP's trim. Example:
string str = trim("  test\r\n\t");

string compact(const string &lan)
Eliminates whitespace characters from the entire string. Example:
string str = compact(" t e s t ");

string str_replace(const string &source, const string &what, const string &dest, TReplacementType rt = RTAll)
Replaces occurrences of a substring in a string with something else. Depending on value of the rt parameters, it can replace all occurrences or only the first or the last occurrence (look up definition of the TReplacementType in my code for details). Inspired by PHP's str_replace. Example:
string str = str_replace("this is a test","is","at");

vector<string> explode(const string &source, const string &separator = "\n")
Converts a string into a vector of strings by breaking the given string at certain spots. The spots are indicated by the separator parameter. Inspired by PHP's explode. Example:
vector<string> vec = explode("Alice\nin\nWonderland");

string implode(const vector<string> &source, const string &separator = "\n")
Merges a vector of strings into a single string by joining the elements of the vector using a given separator string. This operation is opposite to explode.  Inspired by PHP's implode. Example:
string str = implode(vec,"\n");

string htmltostring(const string &source)
Converts piece of HTML code to a text as it is (more or less) displayed by a browser. Example:
std::cout << htmltostring("<b>hey</b> \"<span class=\"q\">Jim</span>\"!");

string htmltocsv(const string &source)
Converts a piece of a HTML code into a string that is ready to be stored in a CSV file format which uses double quotes to delimit strings. Example:
std::cout  << htmltocsv("<b>hey</b> \"<span class=\"q\">Jim</span>\"!");

bool isdigit(char c)
Returns true if the parameter is a digit or false otherwise. Example:
if (isdigit('7')) std::cout << "ok!";

int vecstrfind(const vector<string> &vec, const string &lan)
Returns an index of the vector of strings that points to a given string. Example:
int index = vecstrfind(vec,"Wonderland");

string up(const string& source)
Changes letter case of the string into the upper case. Example:
std::cout << up("Hello World!");

Handling files

bool fileexists(const string &filename)
Checks whether a file with a given file name exists. Example:
if (!fileexists("C:/Windows/System32/kernel32.dll")) std::cout << "oops!";

unsigned int getfilesize(const string &filename)
Returns the size of the file with a given file name. Example:
std::cout << getfilesize("c:/pagefile.sys");

bool download(const string &url, const string &filename)
Downloads a given URL and stores the response in a file. Example:
download("https://en.wikipedia.org/wiki/C%2B%2B","cpp.html");

vector<string> loaddirectory(const string &path)
Searches for files in a directory and returns their list in a vector of string. Wildcard characters are permitted. Example:
vector<string> filez = loaddirectory("c:/windows/system32/*.dll");
for (int i=0;i<filez.size();i++) std::cout << filez[i] << endl;

bool savetofile(const string &content, const string &filename)
Saves string to a file. Example:
savetofile("text,1\nme,2","file.csv");

vector<string> textfile(const string &filename)
Opens a file and reads its content into a vector of strings, each line of the file as a separate string. Inspired by PHP's file. Example:
vector<string> file = textfile("file.csv");
for (int i=0;i<file.size();i++) std::cout << file[i] << endl;

class CCSV
This class wraps a data object that allows for manipulations on a comma delimited file. The file may use (but does not need to) double quotes as text delimiters upon loading. The file will have double quotes as text delimiters upon saving. The data is stored in the data field which is a vector of vectors of strings. Example:
CCSV csv("file.csv");
csv.data[1][1] = "hello";
csv.savetofile("file.csv");

Random numbers

double random()
Generates a real random value uniformly distributed in [0,1). Inspired by PHP's rand. Example:
std::cout << random();

int dice(int minimum, int maximum)
Generates a random integer uniformly distributed between minimum and maximum, inclusive. Inspired by PHP's rand. Example:
std::cout << dice(1,6);

The source code

Download the source code or copy and paste it:

// The file's name is "anutil.h".

#pragma once

#include <sstream>
#include <fstream>
#include <UrlMon.h>
#include <vector>
#include <algorithm>
#include <iomanip>

#pragma comment(lib, "urlmon.lib")

using namespace std;

/* TYPE CONVERSION ROUTINES */

// Converts a string into an integer.
int strtoint(const string &str)  
{
return atoi(str.c_str());
}

// Converts a string object into a wchar_t*.
// Note that this requires manually deleting the string 
// pointer later in the code to avoid memory leaks.
wchar_t* strtowchar(const string &lan)
{
wchar_t *p = new wchar_t[lan.length()+1];
for (int i=0;i<lan.length();i++) p[i] = lan.at(i);
p[lan.length()] = '\0';
return p;
}

// Converts a wstring object into a string object.
string wstrtostr(const wstring &lan)
{
string s(lan.begin(), lan.end());
s.assign(lan.begin(), lan.end());
return s;
}

// Converts an integer value into a string object.
// If the resulting string is shorter than the "length" parameter, 
// then function adds leading zeroes.
string inttostr(int val, int length = 0)
{
stringstream s;
s << val;
string res = s.str();
if (length>res.length())
{
string prefix(length-res.length(),'0');
return prefix+res;
} else return res;
}

// Converts a string object into a real number.
double strtodouble(const string &lan)
{
return atof(lan.c_str());
}

// Converts a real value into a string object.
string doubletostr(double val)
{
stringstream s;
s << setprecision(14) << val;
string res = s.str();
return res;
}

/* STRING PROCESSING ROUTINES */

// Eliminates whitespace characters (space, tab, return, 
// new line) at the beginning and at the end of a string object.
string trim(const string &lan)
{
if (lan.length()==0) return lan;
int i = 0;
while ((lan[i]==' ') || (lan[i]=='\t') || (lan[i]=='\n') || (lan[i]=='\r')) 
{
i++;
if (i==lan.length()) return "";
}
int j = 0;
while ((lan[lan.length()-1-j]==' ') || (lan[lan.length()-1-j]=='\t') 
|| (lan[lan.length()-1-j]=='\n') || (lan[lan.length()-1-j]=='\r')) j++;
return lan.substr(i,lan.length()-i-j);
}

// Eliminates whitespace characters from a entire string.
string compact(const string &lan)
{
string res;
for (int i=0;i<lan.length();i++)
if ((lan[i]!=' ') && (lan[i]!='\t') && (lan[i]!='\n') && (lan[i]!='\r'))
res.push_back(lan[i]);
return res;
}

enum TReplacementType {RTAll, RTFirst, RTLast};

// Replaces occurrences of a substring in a string with something else.
// Depending on value of rt parameters it can replace all occurrences or only 
// first or last occurrence.
string str_replace(const string &source, const string &what, 
const string &dest, TReplacementType rt = RTAll) {
int index = 0;
string res = "";
do
{
int k;
if (rt==RTLast) k = source.rfind(what); else k = source.find(what,index);
if (k<0) return res + source.substr(index);
res += source.substr(index,k-index) + dest;
index = k + what.length();
if (index>=source.length()) return res;
} while (rt==RTAll);
return res + source.substr(index);
}

// Converts a string object into a vector of strings by breaking the given string 
// at certain spots.The spots are given by the separator parameter.
vector<string> explode(const string &source, const string &separator = "\n") {
vector<string> res;
int index = 0;
int k = source.find(separator,index);
while (k>=0)
{
res.push_back(source.substr(index,k-index));
index = k+separator.length();
if (index<source.length()) k = source.find(separator,index); else k = -1;
}
if (index>=source.length()) res.push_back(""); else res.push_back(source.substr(index));
return res;
}

// Merges a vector of strings into a string by joining the elements of a vector 
// with a given separator string. This operation is opposite to explode.
string implode(const vector<string> &source, const string &separator = "\n") {
stringstream s;
for (int i=0;i<source.size();i++) 
{
if (i!=0) s << separator;
s << source[i];
}
return s.str();
}

// This routine converts piece of HTML code to a text as it is 
// (more or less) displayed by a browser.
string htmltostring(const string &source)
{
string lan;
lan = str_replace(source,"&quot;","\"");
lan = str_replace(lan,"&amp;","&");
lan = str_replace(lan,"&#39;","'");
lan = str_replace(lan,"<br />","\n");
lan = str_replace(lan,"&nbsp;"," ");
int lt = lan.find("<");
int gt = lan.find(">",lt);
while ((lt>=0) && (gt>=0))
{
string temp = lan.substr(0,lt);
if (gt+1<lan.size()) temp += " " + lan.substr(gt+1);
lan = temp;
lt = lan.find("<");
gt = lan.find(">",lt);
}
lan = str_replace(lan,"\n"," ");
lan = str_replace(lan,"\r"," ");
lan = str_replace(lan,"\t"," ");
string temp;
if (lan=="") return lan;
temp.push_back(lan[0]);
for (int i=1;i<lan.size();i++)
{
if ((lan[i]==' ') && (temp[temp.size()-1]==' ')) {} else temp.push_back(lan[i]);
}
return temp;
}

// This routine converts a piece of a HTML code into a string that is ready to be 
// stored in a CSV file format which uses double quotes to delimit strings.
string htmltocsv(const string &source)
{
string lan;
lan = htmltostring(source);
lan = str_replace(lan,"\"","\"\"");
lan = "\"" + trim(lan) + "\"";
return lan;
}

// This routine returns true if the parameter is a digit or false otherwise.
bool isdigit(char c)
{
if (((int)c >= 48) && ((int)c <=57)) return true; else return false;
}

// This routine return an index of the vector of strings that points to a given string.
int vecstrfind(const vector<string> &vec, const string &lan)
{
for (int i=0; i<vec.size(); i++) 
{
int w = lan.compare(vec[i]);
if (w==0) return i;
}
return -1;
}

// This routine changes letter case of the string into the upper case.
string up(const string& source)
{
string res = source;
transform(res.begin(),res.end(),res.begin(),::toupper);
return res;
}

/* FILE ROUTINES */

// Checks whether a file with a given filename exists.
bool fileexists(const string &filename)
{
wchar_t* fname = strtowchar(filename);
WIN32_FIND_DATA search_data;
memset(&search_data, 0, sizeof(WIN32_FIND_DATA));
HANDLE handle = FindFirstFile(fname, &search_data);
delete [] fname;
if (handle != INVALID_HANDLE_VALUE) return true; else return false;
}

// Returns the size of the file with a given file name.
unsigned int getfilesize(const string &filename)
{
wchar_t* fname = strtowchar(filename);
WIN32_FIND_DATA search_data;
memset(&search_data, 0, sizeof(WIN32_FIND_DATA));
HANDLE handle = FindFirstFile(fname, &search_data);
delete [] fname;
if (handle != INVALID_HANDLE_VALUE) 
return (search_data.nFileSizeHigh * (MAXDWORD+1)) + search_data.nFileSizeLow;
else return -1;
}

// Downloads a given url and stores the response in a file.
bool download(const string &url, const string &filename)
{
wchar_t filename1[250];
for (int i=0;i<filename.length();i++) filename1[i] = filename.at(i);
filename1[filename.length()] = '\0';
wchar_t url1[250];
for (int i=0;i<url.length();i++) url1[i] = url.at(i);
url1[url.length()] = '\0';
return (URLDownloadToFile(NULL,url1,filename1,0,NULL)==S_OK);
}

// Looks up for files in a directory and returns their list in a vector of string. 
// Wildcard characters are permitted!
vector<string> loaddirectory(const string &path)
{
wchar_t* kupa = strtowchar(path);
WIN32_FIND_DATA search_data;
vector<string> filelist;
memset(&search_data, 0, sizeof(WIN32_FIND_DATA));
HANDLE handle = FindFirstFile(kupa, &search_data); 
delete [] kupa;
while(handle != INVALID_HANDLE_VALUE)
{
filelist.push_back(wstrtostr(search_data.cFileName));
if(FindNextFile(handle, &search_data) == FALSE) break;
}

return filelist;
}

// Saves string to a file. 
bool savetofile(const string &content, const string &filename)
{
ofstream outfile;
outfile.open(filename);
if (!outfile.is_open()) return false;
outfile << content;
outfile.close();
return true;
}

// Opens a file and reads ints content into a vector of strings, 
// each string is a separate line of the file.
vector<string> textfile(const string &filename)
{
ifstream plik(filename,ios::binary);
vector<string> res;
if (!plik.is_open()) return res;
string line;
while (!plik.eof())
{
getline(plik,line);
res.push_back(line);
}
return res;
}

typedef vector<string> TDataRec;
typedef vector<TDataRec> TTable;

// This class represents a data object that allows for manipulations on a comma 
// delimited file with double quotes as text delimiters.
// The data is stored in the TTable object which is a vector of vectors of strings.
class CCSV {
public:
TTable data;
CCSV() {};
CCSV(const string& filename);
bool loadfromfile(const string& filename);
bool savetofile(const string& filename);
};

CCSV::CCSV(const string& filename)
{
loadfromfile(filename);
}

bool CCSV::loadfromfile(const string& filename)
{
ifstream infile(filename);
string line;
while (!infile.eof())
{
getline(infile,line);
if (compact(line)!="") {
int status = 0;  // 0: outside a string; 1: inside a string
string lan = "";
TDataRec rec;
for (int i=0; i<line.size(); i++)
{
switch (status) {
case 0:
{
if (line[i]=='"') 
{
status = 1;
break;
}
if ((line[i]!='"') && (line[i]!=',')) 
{
lan += line[i];
if (i+1==line.size()) 
{
rec.push_back(lan);
lan = "";
} 
break;
}
if (line[i]==',') 
{
rec.push_back(lan);
lan = "";
break;
}
break;
}
case 1:
{
if (line[i]=='"')
{
if ((i+1<line.size()) && (line[i+1]=='"'))
{
i++;
lan += '"';
} else {
i++;
rec.push_back(lan);
lan = "";
status = 0;
}
} else {
lan += line[i];
}
break;
}
}
}
data.push_back(rec);
}
}
infile.close();
return true;
}

bool CCSV::savetofile(const string& filename)
{
ofstream outfile(filename);
for (int i=0; i<data.size(); i++)
{
for (int j=0;j<data[i].size();j++)
{
string lan = "\"" + str_replace(data[i][j],"\"","\"\"") + "\"";
outfile << lan;
if (j+1==data[i].size()) outfile << endl; else outfile << ",";
}
}
outfile.close();
return true;
}

/* RANDOM NUMBER ROUTINES */

// Generates a random real value uniformly distributed in [0,1).
double random()
{
double randmax = (double)(RAND_MAX + 1);
double res = 0;
for (int i=0;i<4;i++) 
{
res += (double)rand();
res = res / randmax;
}
return res;
}

// Generates a random integer uniformly distributed between minimum and maximum, inclusive. 
int dice(int minimum, int maximum)
{
return random()*(maximum-minimum+1)+minimum;
}

More goodies will come.