Exploring with Pandas, Arrow, and the Viewer

~22 min · datasets, explore

Level 0Scout

0 XP0/50 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

Three lenses on the same table

A Dataset is Arrow-backed. You don't need to convert to pandas to peek at it — but if you want pandas, conversion is one call away (ds.to_pandas()) and it's zero-copy if the schema permits. For columnar selection you can also pull a pa.Table with ds.with_format("arrow").

The Hub Viewer

Every dataset page has a Data Studio viewer that auto-queries the Parquet conversion. It's the fastest way to scan a few thousand rows. Behind the scenes it's hitting the Dataset Viewer API, which you can call directly: https://datasets-server.huggingface.co/rows?dataset=...&config=...&split=...&offset=0&length=100. Useful for sampling without ever downloading the dataset.

Code

Pandas + Arrow conversions·python

from datasets import load_dataset

ds = load_dataset("stanfordnlp/imdb", split="train")

# Pandas — convenient for ad-hoc analysis
df = ds.to_pandas()
print(df.label.value_counts())
print(df.text.str.len().describe())

# Arrow — for downstream zero-copy paths
arrow_tbl = ds.with_format("arrow").select(range(10))[:]
print(type(arrow_tbl))  # pyarrow.lib.Table

Hit the dataset viewer API directly·bash

curl -s "https://datasets-server.huggingface.co/rows?dataset=stanfordnlp%2Fimdb&config=plain_text&split=train&offset=0&length=2" | python -m json.tool | head -40

External links

Exercise

Pick three datasets in different domains (text, audio, image). For each, hit the dataset viewer API for the first 50 rows. Save the JSON. Note: which fields are scalar, which are file URLs, which embed binary inline.

Progress

Progress is local-only — sign in to sync across devices.

← PreviousDataset vs DatasetDict, Features, and Schema Next →dataset.map(): Transformations that Cache

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.