Ahmad Al-Shishtawy Senior Researcher, PhD

HDFS Contents Manager for Jupyter Notebooks

I implemented a contents manager for Jupyter notebooks that uses HDFS as a storage backend to store notebooks. I have two versions. The main difference is the library used to read/write HDFS.

The first version uses HDFS3 which is based on libhdfs3, a native C/C++ library to interact with the Hadoop File System (HDFS).
The second version uses Pydoop which is based on the official libhdfs, a JNI based C API for Hadoop’s Distributed File System (HDFS).

The HDFS Contents Manager is used to add Jupyter support to the Hops Big Data platform. Check out the HDFS Contents Manager poster at the SICS Open House 2017 event for more details.

Source Code

The HDFS3 version
The pydoop version

Other articles

OnlineElastMan: Self-Trained Proactive Elasticity Manager for Cloud-Based Storage Services

Published: Fri 18 March 2016
Updated: Tue 11 April 2017
By Ahmad Al-Shishtawy

In Projects.

tags: Elasticity Open Source Cloud Computing

A self-trained procative elasticity manager for the Cloud
read more
SpanEdge: Towards Unifying Stream Processing over Central and Near-the-Edge Data Centers

Published: Tue 01 December 2015
Updated: Thu 08 December 2016
By Ahmad Al-Shishtawy

In Projects.

tags: Edge Computing Open Source Cloud Computing

Unifying Stream Processing over Central and Near-the-Edge Data Centers
read more
ElastMan: Elasticity Manager for Elastic Cloud-Based Services

Published: Mon 09 September 2013
Updated: Mon 16 March 2015
By Ahmad Al-Shishtawy

In Projects.

tags: Elasticity Open Source Cloud Computing

Elasticity Manager for Elastic Cloud-Based Services
read more

Other articles

links

social