O projeto TOFFEE
CASADOCUMENTAÇÃOATUALIZAÇÕESVÍDEOSPESQUISADESCARREGARPATROCINADORESCONTATO


RESEARCH 》 A study on WAN Optimization Techniques

There are various techniques with which one can optimize their WAN Network Data. Any long distance communication can be considered as WAN Network. A decade ago any network connecting two countries, considered as a WAN network, and a network within a city as MAN and soon. But these days in general any long distance communication is considered as WAN Network. Such as your Mobile communication networks, Satellite networks, Space Networks (Deep space networks), Trans-Atlantic cable networks, etc.

You can optimize WAN links with techniques such as:

  • File caching
  • CDN
  • WAN Optimization Devices:
    • Commercial solutions: Riverbed, CISCO, etc
    • or with open-source solution: TOFFEE
  • QoS
  • Compression
  • Data De-duplication

Here is my Youtube video for more in depth coverage of this topic:

CDN Content Delivery Networks (a.k.a Content Distribution Networks): As the name says CDN technology allows you to share your content to large masses. CDN is a way you can scale up your website (or just about any content such as images, videos, live-concert streaming etc). You can build your own private CDN Networks or get can get CDN subscription commercial services via third-party CDN firms such as Akamai, Limelight, etc. I consider CDN as one among the WAN Optimization technologies, since it gives content owners/creators not only an option to distribute their content to masses across the world, but it also provides various benefits such as reliability, speed (faster page loads), etc.

For example Facebook uses Akamai CDN. You can find the proof in the below screenshot of the facebook page view source code:
Facebook using Akamai CDN

And in other case such as Google who built their own private super large CDN Network. In a case like Google they do not rely on third-party commercial CDN providers. Instead they custom built from scratch their content distribution network according to their needs.
Here is my Youtube video for more in depth coverage of this topic:

Data De-duplication: Data de-duplication is a technique by which you can de-duplicate chunks of data (such as packet bytes and packets). De-duplication is a great way to eliminate redundant data. Unlike loss-less compression which is not so effective in all cases, de-duplication helps us save known types of data. Such as known protocols, patterns such as byte sequences with respect to protocol and so on.

For example, in TOFFEE I do packet-wise byte-wise data de-duplication which I call it as Packet Templating feature. It is called Templating since I created thousands of templates for known frequently occurring traffic patterns specific to the packet's application protocol, packet's transport layer protocol (such as TCP/UDP) and so on. It is the Packet templating feature in TOFFEE which is lot more effective saving bandwidth in VoIP sessions, HTTPS (SSL sessions), remote desktop data (such as VNC, Teamviewer), remote MySQL Database connectivity, etc. Here is my Youtube video for more in depth coverage of this topic:

File Caching: File caching is the most oldest and fundamental WAN Optimization technology. One of the best example is caching feature in your browser which does not download pages or resources if they already exist in the browser file cache. And the other best example is your HTTP proxy file cache. Squid-cache is the most popular open-source file caching proxy for the Web supporting HTTP, HTTPS, FTP, and more.

File caching technique reduces bandwidth and improves response times by caching and reusing frequently-requested cache-able web pages (and other file resources such as images, css, java scripts and soon). But unfortunately due to increase in use of dynamic SaaS web applications, file caching is no longer effective in optimizing modern dynamic websites such as Facebook, Youtube, Google-mail, your banking portals, etc. File caching is not applicable for dynamic data such as realtime VoIP, network data-base connections, streaming, Netflix, corporate back-end web-applications/portals and so on. Since the content shared and transferred is not in the form of individual files.

File caching is also useless if you access a file which is not so commonly accessed. The files stored in the cache for certain period of time, eventually it expire and they get purged. So it works purely on luck basis and the real world use-cases are getting limited day by day.

CDN is build with thousands of file-caching servers deployed world-wide. Or in other words CDN is a super-set technology which uses the file-caching technique within its infrastructure. Lets assume you upload a Youtube video, once Google servers processes the same, the video is slowly distributed to various servers (in various geographical datacenters) across the world. And any user residing close to any of these servers may get content from these servers. And if the file does not exist in this server, it may pull the file from the main datacenter (or somewhere), and stores it in the file cache for future use. File caching technique used in CDN is a well scripted predictable choreography. But file caching employed in a generic WAN optimization device is not that effective for the same reasons discussed above. Here is my Youtube video for more in depth coverage of this topic:

Data Compression: Data compression specifically lossless compression is the most common WAN Optimization technique. Although data compression is not so effective for encrypted data (including VPN links), it is still highly effective in optimizing a majority of WAN Network data. If you have VPN links, try to install your WAN Optimization devices (such as TOFFEE), before the VPN devices. This way you can pass the optimized data into your VPN appliances. There are various losses compression algorithms exist and some of the popular ones are: LZ77, LZO, LZ4, LZ4-HC, etc.

TOFFEE supports LZ77, LZO and LZ4. LZO algorithm is enabled as a default kernel compile time option in any pre-compiled TOFFEE kernel binary releases (and OS firmware images) so far (till date 29-Jun-2016).

NASA have their own lossless compression variants and often they are customized. One of the well known algorithms which NASA uses is the LOCO-I (stands for Low Complexity Lossless Compression) which is mainly meant for compressing images. LOCO-I is a kind of lossless compression variant of JPEG. Which is why it is also can be sometimes called as JPEG-LS (stands for JPEG-Lossless). Based on LOCO-I NASA did hardware based solution which is FPGA-LOCO. Since it is hardware based, it is good in performance, reliability and extremely energy efficient.

Apart from this CCSDS have their own variant of RICE lossless compression algorithm.

Read article: A study on Deep Space Networks (DSN) For more information on Deep Space Communication - Space Lossless compression refer my Youtube video below:

For example :: TOFFEE Optimized Satellite ISP Network:
TOFFEE Optimized Satellite ISP Network

Here is my Youtube video for more in depth coverage on this topic:



Suggested Topics:


WAN Optimization and Network Optimization

💎 TOFFEE-MOCHA new bootable ISO: Download
💎 TOFFEE Data-Center Big picture and Overview: Download PDF


Tópicos recomendados:

A study on WAN Optimization Techniques ↗
Saturday' 13-Mar-2021
There are various techniques with which one can optimize their WAN Network Data. Any long distance communication can be considered as WAN Network. A decade ago any network connecting two countries, considered as a WAN network, and a network within a city as MAN and soon. But these days in general any long distance communication is considered as WAN Network. Such as your Mobile communication networks, Satellite networks, Space Networks (Deep space networks), Trans-Atlantic cable networks, etc.

How to check a website using CDN ? ↗
Saturday' 13-Mar-2021

The TOFFEE Project :: TOFFEE-Mocha :: WAN Emulator ↗
Saturday' 13-Mar-2021
The TOFFEE Project :: TOFFEE-Mocha :: Linux Open-Source WAN Emulator

The TOFFEE Project :: TOFFEE-Butterscotch :: Save and Optimize your Internet/WAN bandwidth ↗
Saturday' 13-Mar-2021
TOFFEE-Butterscotch is an open-source software which can be used to save and optimize your Internet/WAN bandwidth. Unlike TOFFEE (and TOFFEE-DataCenter) TOFFEE-Butterscotch is a non peer-to-peer (and asymmetric) network optimization solution. This makes TOFFEE-Butterscotch an ideal tool for all Home and SOHO users.

TOFFEE-Butterscotch Bandwidth saver software development - Update: 28-Oct-2016 ↗
Saturday' 13-Mar-2021
Here is my first software development update of TOFFEE-Butterscotch. In my first TOFFEE-Butterscotch news update I have introduced about TOFFEE-Butterscotch research, project specifications, use-cases, etc. Introducing TOFFEE-Butterscotch Alerts: These are simple packet counters which corresponds to the filter type. For example if the incoming TCP-SYN packets are blocked then its corresponding alert counter will increment whenever such a packet arrives and gets filtered (dropped).

Demo TOFFEE_DataCenter WAN Optimization VM (in VirtualBox) Test Setup ↗
Saturday' 13-Mar-2021
Demo TOFFEE_DataCenter WAN Optimization VM (in VirtualBox) Test Setup



Skype VOIP Data - WAN Acceleration ↗
Saturday' 13-Mar-2021

WAN Optimization - Animated demo of Packet Optimization in TOFFEE-DataCenter ↗
Saturday' 13-Mar-2021

TEST CASES :: TEST RESULTS :: TOFFEE-Mocha-1.0.14 Development version ↗
Saturday' 13-Mar-2021

A study on WAN Optimization Techniques ↗
Saturday' 13-Mar-2021
There are various techniques with which one can optimize their WAN Network Data. Any long distance communication can be considered as WAN Network. A decade ago any network connecting two countries, considered as a WAN network, and a network within a city as MAN and soon. But these days in general any long distance communication is considered as WAN Network. Such as your Mobile communication networks, Satellite networks, Space Networks (Deep space networks), Trans-Atlantic cable networks, etc.



Featured Educational Video:
Assista no Youtube - [8613//1] x254 Kernel Init Code without Kernel Module - Kernel Programming Tip #linode ↗

TOFFEE-Mocha - WAN Emulator :: TOFFEE-MOCHA-2.0.3-0-10-nov-2018-x86-64.iso ↗
Saturday' 13-Mar-2021
Download TOFFEE-MOCHA-2.0.3-0-10-nov-2018-x86-64.iso via Google Drive share: Live bootable x86-64 Debian Stretch 9.5 with light-weight LXDE UI ISO (includes source-code): TOFFEE-MOCHA-2.0.3-0-10-nov-2018-x86-64.iso You can find the source tar-ball in the /root folder. To know more about the project kindly refer TOFFEE- Mocha: News and Updates - Documentation. To know more about current specific release, objectives, features, release notes/updates, quick demo and future road-map, you can watch my video below.

TOFFEE (and TOFFEE-DataCenter) deployment in Large Infrastructure and or ISP Networks ↗
Saturday' 13-Mar-2021
Large Infrastructure or ISP setup: In case if you are an ISP and interested in deploying a large customer WAN Optimized network or an add-on enhanced (WAN Optimized) network for select few customers, then you can deploy something as shown below. Although this case is not meant for hobby/DIY users. This is a feasible solution for high-end professional application and the same can be deployed.

Building my own CDN - Google PageSpeed Insights - Update: 22-Jul-2016 ↗
Saturday' 13-Mar-2021
Ever since after I launched my new The TOFFEE Project website on 1-May'2016, I can see there is a steep increase in traffic. Soon after the launch when I monitored its Alexa rankings it was reporting about 12 Million or so. But once it is getting more and more traffic the Alexa rankings shot up and now currently it shows around 2 Million (as on 22-July-2016). Alexa is an excellent tool to monitor your overall website global ranking and indirectly its performance. Unlike Google Analytics which is bound one or other way into Google's SEO. Alexa gives you a second opinion about your website's growth.

Detect and Monitor Failing Harddrive in Linux - My Seagate 500GB HDD Died ↗
Saturday' 13-Mar-2021
My 500GB Seagate Barracuda 7200RPM hard-drive suddenly started making mild clicking noise. I found this happening since morning. I was suspicious that something wrong in this drive and when I opened the Linux Disks app, I can find the cause of this issue. The disk is increasingly getting read errors. Besides I can see various other parameters such as Power-On Hours, Temperature, Head flying hours, etc.




Why TOFFEE is forked from TrafficSqueezer ↗
Saturday' 13-Mar-2021
TrafficSqueezer is an open-source WAN Optimization project. TrafficSqueezer is mainly a research project which is started around mid-2006. It is initially started as a research (or prototype) code even before it is officially registered in Sourceforge.net. But this code is just primitive user-space raw socket modules. This is later refined and a pre-alpha version is created. Followed by which Alpha release. This prototype code is moved from user-space to Linux Kernel (Kernel Space) and then the journey begin in terms of making a serious WAN Optimization solution. Once the pre-beta and beta releases are complete the mainstream series is started.



Research :: Optimization of network data (WAN Optimization) at various levels:
Network File level network data WAN Optimization


Learn Linux Systems Software and Kernel Programming:
Linux, Kernel, Networking and Systems-Software online classes


Hardware Compression and Decompression Accelerator Cards:
TOFFEE Architecture with Compression and Decompression Accelerator Card [CDN]


TOFFEE-DataCenter on a Dell Server - Intel Xeon E5645 CPU:
TOFFEE-DataCenter screenshots on a Dual CPU - Intel(R) Xeon(R) CPU E5645 @ 2.40GHz - Dell Server