TOFFEE项目
文档更新影片研究下载赞助商联系


RESEARCH 》 Live demo - Data Transfer - High bandwidth to Low bandwidth

I always wanted to do some real experiments and research on packet flow patterns from High-bandwidth to Low-bandwidth networks via networking devices. This is something can be analyzed via capturing Network stack buffer data and other parameters, bench-marking, and so on. But eventually the data-transfer nature and other aspects is often contaminated due to the underlying OS and the way Network stack is implemented. So to understand the nature of packet flow from Higher to Lower bandwidth and vice-versa such as Lower to higher bandwidth, I thought I experiment with various tools and things which physically we can observe this phenomena. What we observe in a software test results is not accurate, Operating system is bound by hardware limitations, bound by non-real-time digitized instruction processing and other latencies. And if we cannot understand this fundamental concept in low-speed networks, then we cannot ever understand its complication in a gigabit or high-speed network. Since the high-speed network processing need even more better hardware and OS capabilities. In this case let us assume everything we are trying to experiment is in OS layer (i.e) not in a dedicated hardware or a complete hardware packet processing platform.
network_research_live_demo_kiran

At times there are cases when packet processing happens in Linux Kernel networking subsystem, at each phase there can be a packet buffer or a packet queue. This "phase" can be a module or a sub-module or a component or individual tiny network stacks and so on. So whenever such a packet queue exists we can assume it almost represents a funnel. Since a funnel will have a interface or guide or hole to pour water/liquid. This is similar to packets getting added to the packet queue. And the funnel has the bottom hole or interface where the liquid pours out (exits) of funnel. This again represents a packet queue where the stored packets are taken out (or fetched) and sent out for further processing. The amount of time liquid spends inside the funnel or the size of funnel more or less represents the packet queue length. It is quite common that we can change these parameters in Linux Kernel via /proc interface.

Suppose if you do the ifconfig command you can find the default txqueuelen parameter.

kiran@desktop-i7-5820k:~$ ifconfig
enx00808e8e90f4 Link encap:Ethernet  HWaddr 00:80:8e:8e:90:f4  
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth0      Link encap:Ethernet  HWaddr fc:aa:14:98:cb:66  
          inet addr:192.168.0.101  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fe80::feaa:14ff:fe98:cb66/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:156020 errors:0 dropped:0 overruns:0 frame:0
          TX packets:104413 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:165547621 (165.5 MB)  TX bytes:15061686 (15.0 MB)
          Interrupt:20 Memory:fb100000-fb120000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:65256 errors:0 dropped:0 overruns:0 frame:0
          TX packets:65256 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:8107654 (8.1 MB)  TX bytes:8107654 (8.1 MB)

In this case the "eth0" interface is my inbuilt motherboard Gigabit Ethernet NIC card. "enx00808e8e90f4" is my USB2 to 100Mbps Ethernet NIC card adapter. And the "lo" is the local host. You can notice the txqueuelen is 1000. txqueuelen is nothing but TX (transmit) queue length. Although as end-users you may see only this in the outside world. But within the Linux kernel you may find packet queues at intermediate levels too. And this almost look like cascading funnels. And each time the funnel sizes varies. This is the reason I took this screenshot of my research video where I am trying to show how they cascade.
network_research_live_demo_kiran_2

Now lets assume you hold a funnel (of average size) as shown and pour water from a tea cup. The capacity of the tea cup is almost the same as funnel's capacity. No matter how fast you pour water, the water never overflows from the top of the funnel brim. This is almost the same case as producer-consumer situation. Where producer-consumer is at the same rate. This is optimal and a good sign.
Note: to highlight the water in the video, I have added red fountain pen ink in the water. So that it is clearly visible.
network_research_live_demo_kiran_3

Now let us imagine we pour water with a large jar as shown in the screenshot. The funnel size is the same, but the water is now being added at a higher rate than what funnel can handle. In this case after a certain point the water will start to overflow from the funnel's top brim. This represents the lack of processing power in your networking device, than the network bandwidth or in other words you are receiving packets at higher rate but you lack the processing power, and at the same time your packet buffers are NOT configured optimally. And the overflow of water from the top of the funnel's brim represents packet drops.

Remember always packet drops happens in a device:

  • anytime if there is congestion
  • if there is data flow from high bandwidth to low-bandwidth
  • some processing delays
  • inadequate buffer capacity (buffer overflow)
  • and so on
network_research_live_demo_kiran_4

In the same way now lets experiment with a small syringe such as Insulin syringe. Insulin syringes have extremely fine needle. And this is a perfect way to obstruct (or add resistance) to liquid flow. As shown in the screenshot each time you such the ink inside the syringe, once you pull the piston, the ink will continue to flow into the syringe although you stopped pulling the piston. This is due to the vacuum created inside and the needle is extremely small to feed the ink quickly. Same way whenever you push the piston, the ink will come out of the syringe slowly and continue to come out of the syringe although you stopped pressing the piston. This is due to the high-pressure inside the syringe and extremely small needle. Infact this is what we can observe in electronic transistors, MOSFETs, vacuum tubes etc dynamically restricting the flow of electrons as well amplifying the electron flow.
network_research_live_demo_kiran_5

So this gives a physical visualization of packet flow in a networking device. Especially cases like networking ports having different bandwidths. As well a case assume you are processing these packets say suppose it is a VPN device, WAN Optimization device, HTTP caching device and so on. The processing of packets adds latency, which means adds resistance to the packet flow. In such cases the Network stack parameters needs to be configured to get optimal performance. Especially the Network packet buffers and so on. And sometimes selecting the right hardware (CPU/RAM/ and so on). So I captured these aspects in a complete video so that my experiments and research may help you to understand these concepts and better architect your Networking components.

Here is the Youtube video link of my complete research:



Suggested Topics:


WAN Optimization and Network Optimization

💎 TOFFEE-MOCHA new bootable ISO: Download
💎 TOFFEE Data-Center Big picture and Overview: Download PDF


推荐主题:

Raspberry Pi as a Networking Device ↗
Saturday' 13-Mar-2021
Raspberry Pi is often used as a single board computer for applications such as IoT, hobby projects, DIY, education aid, research and prototyping device. But apart from these applications Raspberry Pi can be used for real-world applications such as in making a full-fledged networking devices. Raspberry Pi is a single board ARM based hardware which is why it is also classified as ARM based SoC. Since it is ARM based it is highly efficient, tiny form-factor and lower in power consumption with moderate computational power. This will allow it to work several hours on emergency battery backup power supply such as low-cost domestic UPS and or some renewable energy source, which is a prerequisite for a typical networking device.

Streaming CDN Types ↗
Saturday' 13-Mar-2021

TOFFEE deployment topology guide ↗
Saturday' 13-Mar-2021
Assume you have two sites (such as Site-A and Site-B) connected via slow/critical WAN link as shown below. You can optimize this link by saving the bandwidth as well possibly improve the speed. However, the WAN speed can be optimized only if the WAN link speeds are below that of the processing latency of your TOFFEE installed hardware. Assume your WAN link is 12Mbps, and assume the maximum WAN optimization speed/capacity of Raspberry Pi is 20Mbps, then your link will get speed optimization too. And in another case, assume your WAN link is 50Mbps, then using the Raspberry Pi as WAN Optimization device will actually increase the latency (i.e slows the WAN link). But in all the cases the bandwidth savings should be the same irrespective of the WAN link speed. In other words, if you want to cut down the WAN link costs via this WAN Optimization set up, you can always get it since it reduces the overall bandwidth in almost all the cases (including encrypted and pre-compressed data).

TOFFEE Data-Center WAN Optimization deployment in Big Data Analytics ↗
Saturday' 13-Mar-2021

TOFFEE-DataCenter WAN Optimization - Google Hangouts Demo and VOIP Optimization ↗
Saturday' 13-Mar-2021

TOFFEE (and TOFFEE-DataCenter) deployment in Large Infrastructure and or ISP Networks ↗
Saturday' 13-Mar-2021
Large Infrastructure or ISP setup: In case if you are an ISP and interested in deploying a large customer WAN Optimized network or an add-on enhanced (WAN Optimized) network for select few customers, then you can deploy something as shown below. Although this case is not meant for hobby/DIY users. This is a feasible solution for high-end professional application and the same can be deployed.

在YouTube上观看 - [465//1] 158 VLOG - TOFFEE WAN Optimization Software Development live update - 6-Nov-2016 ↗


CDN Hosting ↗
Saturday' 13-Mar-2021
It is quite interesting that there are few web hosting firms are offering direct CDN based hosting services. Since it is a direct CDN based hosting, it is cheap, extremely easy or transparent CDN service. It is transparent, since each time you publish your content in the hosting web-server (origin server), it is immediately is in sync automatically in the user-serving CDN caching machines. Since the hosting vendor and the CDN vendor are all the same, it is also easy to use their services. There is no incompatibility issues, interoperability issues, and better integrated analytics, are all the benefits of CDN Hosting services.

CDN Content Delivery Networks - Types ↗
Saturday' 13-Mar-2021

TOFFEE-DataCenter screenshots on a Dual CPU - Intel(R) Xeon(R) CPU E5645 @ 2.40GHz - Dell Server ↗
Saturday' 13-Mar-2021

My sample Wireshark packet capture files for research ↗
Saturday' 13-Mar-2021
I have a huge repository (or collection) of sample Wireshark packet capture files for reference. I use them extensively for research and development of TOFFEE as well to understand various protocol PDUs and protocol standards. I personally collected various test captures via Wireshark during my test and experimental research setup during the course of TOFFEE development. Say if you are a student and learning Networking and or say VoIP data and VoIP packets, you can analyse my VoIP sample Wireshark captures. Or in other case assume you are doing some quick research (or development) and want to refer few handful of VoIP packets then you can download and analyse my sample packet capture files.



Featured Educational Video:
在YouTube上观看 - [170//1] 169 Q&A - Add additional HardDrive or storage space in Linux VirtualBox VM ↗

Tracking Live Network Application Data - in a WAN Acceleration (WAN Optimization) Device ↗
Saturday' 13-Mar-2021

TOFFEE with Hardware Compression and Decompression Accelerator Cards ↗
Saturday' 13-Mar-2021
You can build a basic TOFFEE WAN Optimization hardware completely in software layer (i.e its networking data-plane and control-plane). And if you are a product manufacturer you can make commercial WAN Optimization products with TOFFEE with software layer alone. And if you choose to improve its performance, you can use any third-party PCIe Compression Accelerator cards.

Moon Base and Space Colonization - First we need fast InterPlanetary Internet ↗
Saturday' 13-Mar-2021

TOFFEE Download :: TOFFEE-1.1.70-1-portable ↗
Saturday' 13-Mar-2021



在YouTube上观看 - [1882//1] Deep Space Communication - Episode1 - Introduction ↗

TOFFEE-Mocha WAN Emulation software development - Update: 18-June-2016 ↗
Saturday' 13-Mar-2021
In the previous update (17-Jun-2016) I discussed about the upcoming new Random Packet drop feature along with other completed features. Now I completed the entire TOFFEE-Mocha Random packet drop feature. I completed all the kernel components and the UI support of the same. And to make GUI settings more organized I split the earlier Basic-Settings page into two separate pages namely: Packet Drop and Packet Delay. So this way it is simple to understand settings according to their functionality.



Research :: Optimization of network data (WAN Optimization) at various levels:
Network File level network data WAN Optimization


Learn Linux Systems Software and Kernel Programming:
Linux, Kernel, Networking and Systems-Software online classes


Hardware Compression and Decompression Accelerator Cards:
TOFFEE Architecture with Compression and Decompression Accelerator Card [CDN]


TOFFEE-DataCenter on a Dell Server - Intel Xeon E5645 CPU:
TOFFEE-DataCenter screenshots on a Dual CPU - Intel(R) Xeon(R) CPU E5645 @ 2.40GHz - Dell Server