Ways to create huge lists of incrementing data

[Allen]

Hi all,

I need to create some text files with billions of lines of data along the lines of the following:

Data1 Value1
Data2 Value2
...
Data1000000000 Value1000000000

I used to use Excel for this sort of task but have never gone above it's limit before (Excel 2010 so 1,048,576 rows), and I cannot seem to find anything on Google that'll help.

I am not a developer, but don't have a complete lack of programming knowledge and I know that'll this could easily be done in a simple bash script for example, but I don't know how to do it.

Any ideas chaps (or chappettes of course )?

[Jonatron]

I'd just write a quick python script. Do you really mean billions? You'll end up with quite a big file. eg:

C:\Users\Jon>python
Python 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on win
32
Type "help", "copyright", "credits" or "license" for more information.
>>> f = open("foo.bar", "w")
>>> for i in xrange(1, 10):
...     f.write("Data%s Value%s" % (i, i))
...
>>> f.close()
>>>

This would give you a file named foo.bar containing:

Data1 Value1
Data2 Value2
Data3 Value3
Data4 Value4
Data5 Value5
Data6 Value6
Data7 Value7
Data8 Value8
Data9 Value9

[TheAnimus]

I am curious, what is this for?

[Allen]

Flooding a DB server with data to check the hardware reliability. We recently had two identical servers with 4 identical drives "fail" and we are trying to find out why. The only difference between these and many other servers we have is one piece of software (a DB server that we hadn't heard of before) that is only on these servers.

Thanks Jon, will give that a go!

[wasabi]

Windows lazy bodger approach. Create a 3 line batch file

:start
dir c: >>output.txt
goto start

[TheAnimus]

Could easily flood the database with a piece of SQL mate!

DECLARE @i int = 0
WHILE @i < 2000000 BEGIN
   INSERT INTO Foo Values (i,'This is some test data, tralalalalallalalalalallalalalalalla')
END

Kids, don't go using while loops on PHP websites to see if they sanitise their inputs, its not cool.

[Allen]

Thanks, but it's not SQL, otherwise I would written that myself.

(Should've said in the OP, it's Redis, never used it before)

[TheAnimus]

Ah ic!

Redis is fun little thing isn't it. It's effectively a fancy file system, rather than a DB, its not a RDBMS.

If you are really wanting to test it, I'd recommend checking out their page:
http://redis.io/clients
and making something to insert in to it at a little bit of a pseudo random threshold, I've not looked, but you might find something like Netflix's chaos monkey, which will provide non-sequential insertion a much better test.

[Allen]

Cheers, I'll check it out!