Results 1 to 9 of 9

Thread: Ways to create huge lists of incrementing data

  1. #1
    HEXUS.social member Allen's Avatar
    Join Date
    Nov 2003
    Location
    ::1
    Posts
    8,530
    Thanks
    363
    Thanked
    261 times in 167 posts
    • Allen's system
      • Motherboard:
      • Asus MAXIMUS VIII GENE
      • CPU:
      • Intel® Core™ i5-6600K @ 3.5GHz
      • Memory:
      • Kingston HyperX Predator DDR4 2x8GB
      • Storage:
      • 256GB Samsung 960 PRO + 512GB Samsung 960 Evo
      • Graphics card(s):
      • Asus ROG Strix GeForce GTX 1080 Ti
      • PSU:
      • XFX P1-650X-NLG9 XXX 650W Modular
      • Case:
      • Fractal Design Node 804
      • Operating System:
      • Windows 10 Home 64-bit
      • Monitor(s):
      • 27" BenQ XL2730Z + 23" Dell U2311H
      • Internet:
      • Virgin Media 200Mbps

    Ways to create huge lists of incrementing data

    Hi all,

    I need to create some text files with billions of lines of data along the lines of the following:

    Data1 Value1
    Data2 Value2
    ...
    Data1000000000 Value1000000000

    I used to use Excel for this sort of task but have never gone above it's limit before (Excel 2010 so 1,048,576 rows), and I cannot seem to find anything on Google that'll help.

    I am not a developer, but don't have a complete lack of programming knowledge and I know that'll this could easily be done in a simple bash script for example, but I don't know how to do it.

    Any ideas chaps (or chappettes of course )?

  2. #2
    Almost Ex-HEXUS Staff Jonatron's Avatar
    Join Date
    Sep 2009
    Location
    London
    Posts
    549
    Thanks
    45
    Thanked
    157 times in 101 posts

    Re: Ways to create huge lists of incrementing data

    I'd just write a quick python script. Do you really mean billions? You'll end up with quite a big file. eg:
    Code:
    C:\Users\Jon>python
    Python 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on win
    32
    Type "help", "copyright", "credits" or "license" for more information.
    >>> f = open("foo.bar", "w")
    >>> for i in xrange(1, 10):
    ...     f.write("Data%s Value%s" % (i, i))
    ...
    >>> f.close()
    >>>
    This would give you a file named foo.bar containing:
    Data1 Value1
    Data2 Value2
    Data3 Value3
    Data4 Value4
    Data5 Value5
    Data6 Value6
    Data7 Value7
    Data8 Value8
    Data9 Value9

  3. Received thanks from:

    Allen (25-05-2013)

  4. #3
    Seething Cauldron of Hatred TheAnimus's Avatar
    Join Date
    Aug 2005
    Posts
    17,147
    Thanks
    798
    Thanked
    2,151 times in 1,407 posts

    Re: Ways to create huge lists of incrementing data

    I am curious, what is this for?
    throw new ArgumentException (String, String, Exception)

  5. #4
    HEXUS.social member Allen's Avatar
    Join Date
    Nov 2003
    Location
    ::1
    Posts
    8,530
    Thanks
    363
    Thanked
    261 times in 167 posts
    • Allen's system
      • Motherboard:
      • Asus MAXIMUS VIII GENE
      • CPU:
      • Intel® Core™ i5-6600K @ 3.5GHz
      • Memory:
      • Kingston HyperX Predator DDR4 2x8GB
      • Storage:
      • 256GB Samsung 960 PRO + 512GB Samsung 960 Evo
      • Graphics card(s):
      • Asus ROG Strix GeForce GTX 1080 Ti
      • PSU:
      • XFX P1-650X-NLG9 XXX 650W Modular
      • Case:
      • Fractal Design Node 804
      • Operating System:
      • Windows 10 Home 64-bit
      • Monitor(s):
      • 27" BenQ XL2730Z + 23" Dell U2311H
      • Internet:
      • Virgin Media 200Mbps

    Re: Ways to create huge lists of incrementing data

    Flooding a DB server with data to check the hardware reliability. We recently had two identical servers with 4 identical drives "fail" and we are trying to find out why. The only difference between these and many other servers we have is one piece of software (a DB server that we hadn't heard of before) that is only on these servers.

    Thanks Jon, will give that a go!

  6. #5
    Account closed at user request
    Join Date
    Aug 2003
    Location
    Elephant watch camp
    Posts
    2,150
    Thanks
    56
    Thanked
    115 times in 103 posts
    • wasabi's system
      • Motherboard:
      • MSI B85M-G43
      • CPU:
      • i3-4130
      • Memory:
      • 8 gig DDR3 Crucial Rendition 1333 - cheap!
      • Storage:
      • 128 gig Agility 3, 240GB Corsair Force 3
      • Graphics card(s):
      • Zotac GTX 750Ti
      • PSU:
      • Silver Power SP-S460FL
      • Case:
      • Lian Li T60 testbanch
      • Operating System:
      • Win7 64bit
      • Monitor(s):
      • First F301GD Live
      • Internet:
      • Virgin cable 100 meg

    Re: Ways to create huge lists of incrementing data

    Windows lazy bodger approach. Create a 3 line batch file

    :start
    dir c: >>output.txt
    goto start

  7. #6
    Seething Cauldron of Hatred TheAnimus's Avatar
    Join Date
    Aug 2005
    Posts
    17,147
    Thanks
    798
    Thanked
    2,151 times in 1,407 posts

    Re: Ways to create huge lists of incrementing data

    Could easily flood the database with a piece of SQL mate!

    Code:
    DECLARE @i int = 0
    WHILE @i < 2000000 BEGIN
       INSERT INTO Foo Values (i,'This is some test data, tralalalalallalalalalallalalalalalla')
    END
    Kids, don't go using while loops on PHP websites to see if they sanitise their inputs, its not cool.
    throw new ArgumentException (String, String, Exception)

  8. Received thanks from:

    Allen (25-05-2013)

  9. #7
    HEXUS.social member Allen's Avatar
    Join Date
    Nov 2003
    Location
    ::1
    Posts
    8,530
    Thanks
    363
    Thanked
    261 times in 167 posts
    • Allen's system
      • Motherboard:
      • Asus MAXIMUS VIII GENE
      • CPU:
      • Intel® Core™ i5-6600K @ 3.5GHz
      • Memory:
      • Kingston HyperX Predator DDR4 2x8GB
      • Storage:
      • 256GB Samsung 960 PRO + 512GB Samsung 960 Evo
      • Graphics card(s):
      • Asus ROG Strix GeForce GTX 1080 Ti
      • PSU:
      • XFX P1-650X-NLG9 XXX 650W Modular
      • Case:
      • Fractal Design Node 804
      • Operating System:
      • Windows 10 Home 64-bit
      • Monitor(s):
      • 27" BenQ XL2730Z + 23" Dell U2311H
      • Internet:
      • Virgin Media 200Mbps

    Re: Ways to create huge lists of incrementing data

    Thanks, but it's not SQL, otherwise I would written that myself.

    (Should've said in the OP, it's Redis, never used it before)

  10. #8
    Seething Cauldron of Hatred TheAnimus's Avatar
    Join Date
    Aug 2005
    Posts
    17,147
    Thanks
    798
    Thanked
    2,151 times in 1,407 posts

    Re: Ways to create huge lists of incrementing data

    Ah ic!

    Redis is fun little thing isn't it. It's effectively a fancy file system, rather than a DB, its not a RDBMS.

    If you are really wanting to test it, I'd recommend checking out their page:
    http://redis.io/clients
    and making something to insert in to it at a little bit of a pseudo random threshold, I've not looked, but you might find something like Netflix's chaos monkey, which will provide non-sequential insertion a much better test.
    throw new ArgumentException (String, String, Exception)

  11. Received thanks from:

    Allen (25-05-2013)

  12. #9
    HEXUS.social member Allen's Avatar
    Join Date
    Nov 2003
    Location
    ::1
    Posts
    8,530
    Thanks
    363
    Thanked
    261 times in 167 posts
    • Allen's system
      • Motherboard:
      • Asus MAXIMUS VIII GENE
      • CPU:
      • Intel® Core™ i5-6600K @ 3.5GHz
      • Memory:
      • Kingston HyperX Predator DDR4 2x8GB
      • Storage:
      • 256GB Samsung 960 PRO + 512GB Samsung 960 Evo
      • Graphics card(s):
      • Asus ROG Strix GeForce GTX 1080 Ti
      • PSU:
      • XFX P1-650X-NLG9 XXX 650W Modular
      • Case:
      • Fractal Design Node 804
      • Operating System:
      • Windows 10 Home 64-bit
      • Monitor(s):
      • 27" BenQ XL2730Z + 23" Dell U2311H
      • Internet:
      • Virgin Media 200Mbps

    Re: Ways to create huge lists of incrementing data

    Cheers, I'll check it out!

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •