Dask Reviews and details

Social recommendations and mentions

• Large Scale Hydrology: Geocomputational tools that you use

  We're using a lot of Python. In addition to these, gridMET, Dask, HoloViz, and kerchunk. Source: about 2 years ago

• msgspec - a fast & friendly JSON/MessagePack library

  I wrote this for speeding up the RPC messaging in dask, but figured it might be useful for others as well. The source is available on github here: https://github.com/jcrist/msgspec. Source: about 2 years ago

• What does it mean to scale your python powered pipeline?

  Dask: Distributed data frames, machine learning and more. - Source: dev.to / over 2 years ago

• Data pipelines with Luigi

  To do that, we are efficiently using Dask, simply creating on-demand local (or remote) clusters on task run() method:. - Source: dev.to / over 2 years ago

• How to load 85.6 GB of XML data into a dataframe

  I’m quite sure dask helps and has a pandas like api though will use disk and not just RAM. Source: over 2 years ago

• How to load 85.6 GB of XML data into a dataframe

  You'll need to use a tool that allows you to apply the kind of operations you're trying to do over chunks of data; dask comes to mind as an option. Source: over 2 years ago

• Show HN: Hamilton, a Microframework for Creating Dataframes

  This project reminds me a lot of Dask https://dask.org/. A library that allows delayed calculation of complex dataframes in Python. - Source: Hacker News / over 2 years ago

• FUNCTOOLS CHANGED MY LIFE

  I don’t do much ETL/System Integration lately, but I also keep an eye on another impressive library: Dask (https://dask.org). Source: over 2 years ago

• Are there any fast alternatives to databases (for tabular data but without SQL)?

  I haven't used parquet from C++ yet, but I have done some data analysis in python with dask dataframes, where I used parquet as a file storage format. Dask abstracts the iteration of chunks away. But I'm certain this is also possible with C++. Source: over 2 years ago

• Spark DF’s vs Pandas DF’s

  You can also check out Dask if you eventually need to or want to run clustered pandas. Source: over 2 years ago

• Boss wants 17 billion rows put into Pandas. Can it do that?

  You can try using Dask. Its very similar to Panda's syntax but meant to handle big data when Pandas beings to struggle due to memory. Found this Medium post that gives an overview of it. Here is the link to Dask's documentation. Source: almost 3 years ago

• How to release the memory of dataframe ?

  If this still isn't enough, look into using dask, essentially it's designed to allow you to work with pandas data frames when the size of the data is bigger than your computer's memory. Source: almost 3 years ago

• Putting Cassandra driver output into pandas dataframe

  You can’t if it’s memory error. There are other solutions out there which distributes. For example, if your computer only has 16gb RAM and your dataset is 30gb then you can’t load it all at once in pandas. Try dask instead. Source: almost 3 years ago

• Can someone help me to understand async please

  Not async though with pandas performance issues you may want to try dask. Source: almost 3 years ago

• Hyperparameter tuning in multiple/sequential slurm jobs?

  Dask has integration with resource management systems like slurm and openpbs. It provides a client and scheduling abstraction so that you can code to the abstraction without having to care about whether your code is going to be run on a single machine, a cloud system, or a large HPC cluster. Dask-jobqueue allows you to readily launch jobs for slurm, making the requested resources available to the Dask scheduler... Source: almost 3 years ago

• Why is Python popular despite being accused of being slow?

  Not everyone has the same "parallelism" needs. I have used mpi4py to distribute scientific computations using numpy over thousands of cores on hundreds of servers with much less effort than doing the same thing in C / C++ and almost no performance penalty (I could batch my data in big enough chunks). Today there are higher level distributed computing packages like dask that are even easier to use. Source: about 3 years ago