dragonet
5 years agoexternal: add licence master release2015-06-08
Kornilios Kourtis [Sun, 7 Jun 2015 09:06:01 +0000]
external: add licence

5 years agoDragonet: add licence
Kornilios Kourtis [Sun, 7 Jun 2015 08:52:34 +0000]
Dragonet: add licence

5 years agoremove unused scripts/qsmsg.sh
Kornilios Kourtis [Sun, 7 Jun 2015 09:13:49 +0000]
remove unused scripts/qsmsg.sh

5 years agocomments/misc changes
Kornilios Kourtis [Thu, 28 May 2015 08:33:04 +0000]
comments/misc changes

5 years agominor improvements in the benchmarking script to make sure that
Pravin [Wed, 27 May 2015 07:16:59 +0000]
minor improvements in the benchmarking script to make sure that
echoserver benchmark works with properly.

5 years agoAdded a simple script which will cleanup everything and recompile
Pravin Shinde [Wed, 27 May 2015 07:29:30 +0000]
Added a simple script which will cleanup everything and recompile

5 years agocreating a special script to simplify setting up of babybel2 from it's
Pravin [Wed, 27 May 2015 07:25:06 +0000]
creating a special script to simplify setting up of babybel2 from it's
harddisk repository.

5 years agoAdded support for running with 9 queues,
Pravin Shinde [Sun, 24 May 2015 23:56:24 +0000]
Added support for running with 9 queues,
recording the server pid for more reliable startup,
extended the hack about dealing with slow consumer

5 years agoAdded the details of babybel4 into the setIPaddress script
Pravin [Thu, 21 May 2015 19:34:13 +0000]
Added the details of babybel4 into the setIPaddress script

5 years agomodified the data-collections scripts to suit new changes.
Pravin [Thu, 21 May 2015 19:30:14 +0000]
modified the data-collections scripts to suit new changes.
Updated plotting scripts to work with new setup
Added a wrapper around memaslap which simplifies running it from
the compiled directory by linking to the proper libraries.

5 years agoAdded support to prepare machine script to link with the tools
Pravin [Thu, 21 May 2015 19:24:06 +0000]
Added support to prepare machine script to link with the tools
over NFS (for quick client machines setup on ETHZ setup)

5 years ago - changes to the dragonet code to make running onload library more robust
Pravin Shinde [Thu, 21 May 2015 19:20:51 +0000]
 - changes to the dragonet code to make running onload library more robust
 - disabled dropping invalid UDP checksum packets to work around SF issue
 - disabled an extra lock in the dragonet-onload driver  as we are using
       same thread to send and receive packets on a single queue.
 - increased memory for bulk transfer queues
 - Made sbrinz1/2 as high priority clients (as ziger2 had issues about
    currupt packets)

5 years agoadapted the benchmarking scripts to reflect new name of stack-e10k-dpdk
Pravin [Sun, 10 May 2015 23:00:18 +0000]
adapted the benchmarking scripts to reflect new name of stack-e10k-dpdk

5 years agoimproved the documentation of dpdk install readme,
Pravin Shinde [Sun, 10 May 2015 23:01:40 +0000]
improved the documentation of dpdk install readme,
and some cleanup wiht custom fancyecho

5 years agosf NIC is working with newer interface of search in dragonet.
Pravin Shinde [Sun, 10 May 2015 22:49:46 +0000]
sf NIC is working with newer interface of search in dragonet.
Also, added some work-arounds to avoid udp checksum issue

5 years agorefactor/document Search.hs
Kornilios Kourtis [Thu, 2 Apr 2015 12:17:27 +0000]
refactor/document Search.hs

5 years agoupdate README
Kornilios Kourtis [Wed, 1 Apr 2015 13:34:22 +0000]
update README

5 years agoupdate build scripts
Kornilios Kourtis [Wed, 1 Apr 2015 13:30:33 +0000]
update build scripts

5 years agoremove deprecated stack-dpdk.hs
Kornilios Kourtis [Tue, 31 Mar 2015 15:26:52 +0000]
remove deprecated stack-dpdk.hs

stack-dpdk2 is renamed to stack-e10k-dpdk
stack-e10k  is renamed to stack-e10k-base

Both binaries are produced from stack-e10k.hs.

5 years agoremove old/unused e10k/e1000 dirs
Kornilios Kourtis [Tue, 31 Mar 2015 14:30:47 +0000]
remove old/unused e10k/e1000 dirs

5 years agocabal haddock now works
Kornilios Kourtis [Tue, 31 Mar 2015 12:33:02 +0000]
cabal haddock now works

5 years agoRemoved the older version of dpdk that we are not using anymore
Pravin Shinde [Mon, 30 Mar 2015 09:06:44 +0000]
Removed the older version of dpdk that we are not using anymore

5 years agoRemoved the scenario folder which was used to document possible
Pravin Shinde [Mon, 30 Mar 2015 09:05:49 +0000]
Removed the scenario folder which was used to document possible
usecases for the first version of dragonet

5 years agoremoved PRG folder which contained first version of PRG files
Pravin Shinde [Mon, 30 Mar 2015 09:04:59 +0000]
removed PRG folder which contained first version of PRG files
which are not relavent anymore

5 years agoAdded README with installation instructions for Dragonet.
Pravin [Mon, 9 Mar 2015 16:23:59 +0000]
Added README with installation instructions for Dragonet.
Also updated the installation script to make it more readable.
The installation script will make sure that you will install proper
version of cabal-install which will not conflict with llvm-gen.
Modified dpdk install script to automate steps and document them better.
Added cabal.config generated with `cabal freeze` (also older working
version is kept in Dragonet/scripts/pravin/cabal.config_old for ref).

5 years agorenamed the dpdk_helpers folder to more appropriate install_helpers
Pravin Shinde [Thu, 5 Mar 2015 16:02:31 +0000]
renamed the dpdk_helpers folder to more appropriate install_helpers

5 years agoMerge new search/flowmap algorithm
Kornilios Kourtis [Tue, 17 Feb 2015 11:09:59 +0000]
Merge new search/flowmap algorithm

Conflicts:
Dragonet/tests/search.hs

5 years agoupdated the harness scripts to work with fancyechoLinux.
Pravin [Tue, 3 Feb 2015 15:37:17 +0000]
updated the harness scripts to work with fancyechoLinux.

5 years agominor changes to support priority classification for echoserver traffic
Pravin Shinde [Tue, 3 Feb 2015 15:37:06 +0000]
minor changes to support priority classification for echoserver traffic

5 years agoadded support in SF for incremental oracle, and made it work with
Pravin Shinde [Mon, 2 Feb 2015 15:29:02 +0000]
added support in SF for incremental oracle, and made it work with
new interace of stack-e10k style.

5 years agobenchSearch updates
Kornilios Kourtis [Mon, 2 Feb 2015 09:50:35 +0000]
benchSearch updates

5 years agoflowmap: fmUpdateNodeReplace_ and tweaks
Kornilios Kourtis [Sun, 1 Feb 2015 20:38:28 +0000]
flowmap: fmUpdateNodeReplace_ and tweaks

5 years agosearch/qmap: replacment checkpoint
Kornilios Kourtis [Sun, 1 Feb 2015 14:44:45 +0000]
search/qmap: replacment checkpoint

There are still problems with jump.

5 years agosearch: fix allQueues_
Kornilios Kourtis [Sun, 1 Feb 2015 14:43:38 +0000]
search: fix allQueues_

5 years agomoved old and unwanted benchmarking scripts in separate directory to
Pravin [Sun, 1 Feb 2015 10:42:39 +0000]
moved old and unwanted benchmarking scripts in separate directory to
as a cleanup

5 years agosome more streamlining of plotting script
Pravin Shinde [Sun, 1 Feb 2015 10:21:35 +0000]
some more streamlining of plotting script

5 years agoimported and modified the plotting scripts to do online workload plots
Pravin Shinde [Sat, 31 Jan 2015 09:32:22 +0000]
imported and modified the plotting scripts to do online workload plots

5 years agoModified scripts to record per second stats for memaslap
Pravin [Sun, 1 Feb 2015 10:16:20 +0000]
Modified scripts to record per second stats for memaslap
 - Modified harness scripts to be able to add dynamic flows later
    They can also run all flows together
 - Added support in harness to reliably detect when other clients have
   actually started their benchmarking phase

5 years agosearch/flowmap: checkpoint for replacing ccs
Kornilios Kourtis [Sun, 1 Feb 2015 09:40:43 +0000]
search/flowmap: checkpoint for replacing ccs

Simple example seems to work

5 years agoAdded a testcase for priority cost function which breaks for 5 queue
Pravin Shinde [Sun, 1 Feb 2015 10:09:36 +0000]
Added a testcase for priority cost function which breaks for 5 queue
    It is in test_incr_pravin_testcase_mixed
 - Added optimization to memcached to group newly arriving flows and
   to trigger dragonet-optimization phase in batch

5 years agosearch/flowmap: explicitly maintan cc/flow association
Kornilios Kourtis [Sat, 31 Jan 2015 12:55:43 +0000]
search/flowmap: explicitly maintan cc/flow association

5 years agoflowmap: rmFlow
Kornilios Kourtis [Sat, 31 Jan 2015 11:52:19 +0000]
flowmap: rmFlow

5 years agoflowmap: notes
Kornilios Kourtis [Sat, 31 Jan 2015 10:13:30 +0000]
flowmap: notes

5 years agotests/BalanceAcrossRxQs updates
Kornilios Kourtis [Sat, 31 Jan 2015 10:08:24 +0000]
tests/BalanceAcrossRxQs updates

5 years agoconf/flowmap: pass conf state to flow map
Kornilios Kourtis [Sat, 31 Jan 2015 10:07:16 +0000]
conf/flowmap: pass conf state to flow map

5 years agoconf: add modified nodes to conf state
Kornilios Kourtis [Sat, 31 Jan 2015 09:27:30 +0000]
conf: add modified nodes to conf state

This increases incremental search for 100 flows from .85sec to ~1sec.

5 years agominor fix to avoid queue zero from allocation
Pravin Shinde [Fri, 30 Jan 2015 23:42:02 +0000]
minor fix to avoid queue zero from allocation

5 years agoHere is the list of changes in this commit
Pravin Shinde [Fri, 30 Jan 2015 22:17:01 +0000]
Here is the list of changes in this commit
 - The current graph is saved in StackState, and reused for future
   socket level changes
 - Currently new graph generation and reduction happens only after
   version 4 (wherer first unplanned flow is suppsoed to come)
 - Added doTimeIt at few interesting places
 - Removed lot of debug prints so that they will not mess with doTimeIt
 - made sure that queue-0 goes to socket-0

5 years agostatic cost function
Kornilios Kourtis [Fri, 30 Jan 2015 21:25:47 +0000]
static cost function

5 years agoAdded support for running multiple servers together with dragonet stack
Pravin [Fri, 30 Jan 2015 13:30:33 +0000]
Added support for running multiple servers together with dragonet stack
and benchmark them either together or in isolation.
  - Currently the costfunction used is hardcoded in dragonet_stack.py
  - The decision to start the stack is inferred by overloaded meaning of
      SERVER_CORESHIFT parameter (3 == don't start stack, else start
      stack)

5 years agoChanges done while benchmarking multi-application setup
Pravin Shinde [Fri, 30 Jan 2015 13:21:37 +0000]
Changes done while benchmarking multi-application setup
 - Q0 is not used in allocation
 - Q0 is exclusively mapped to one of the socket from app
        (TODO: needs to be first socket)
 - Ability to control queue to socket mappping
 - Ability to control flow detection in memcached
 - Additional thread to handle queue0 in memcached
 - More readable queue mappings in "stack.filtready" log file

5 years agosearch: tweaks
Kornilios Kourtis [Thu, 29 Jan 2015 14:33:19 +0000]
search: tweaks

5 years agopriorityCost: do not allow flows on non-exclusive queues
Kornilios Kourtis [Thu, 29 Jan 2015 11:57:22 +0000]
priorityCost: do not allow flows on non-exclusive queues

5 years agominor change to make my test compatible with kornilios's new interface
Pravin Shinde [Thu, 29 Jan 2015 11:19:30 +0000]
minor change to make my test compatible with kornilios's new interface
for incremental oracle

5 years agoAdded another testcase to show the problem with priority workload
Pravin Shinde [Thu, 29 Jan 2015 11:07:45 +0000]
Added another testcase to show the problem with priority workload

5 years agosearch: add incremental configuration as a result
Kornilios Kourtis [Thu, 29 Jan 2015 09:30:42 +0000]
search: add incremental configuration as a result

5 years agoMore streamlining of benchmarking scripts for echoserver.
Pravin [Thu, 29 Jan 2015 02:57:32 +0000]
More streamlining of benchmarking scripts for echoserver.
Now able to run both memcached and echoserver together with clients as
well (changes in data_collection.sh file with '-b' switch)

5 years agoModified search to make incremenal oracle use fdir filters instead of
Pravin Shinde [Thu, 29 Jan 2015 02:58:29 +0000]
Modified search to make incremenal oracle use fdir filters instead of
5tuple filters

5 years agoUpdated the benchmarking scripts to work with new setup.
Pravin [Wed, 28 Jan 2015 23:12:50 +0000]
Updated the benchmarking scripts to work with new setup.
Also, updated the echoserver benchmarking code which was rusted a bit.

5 years agoScript changes to handle new parameters to stack and new way to detect
Pravin Shinde [Wed, 28 Jan 2015 23:01:20 +0000]
Script changes to handle new parameters to stack and new way to detect
that application is ready.  Now we look into "stack.filtready" file
to see how many filters are inserted, and that gives us notion of how
much of work is already done.

5 years agoNow creating the logs about filter manipulation into file so that
Pravin Shinde [Wed, 28 Jan 2015 14:25:52 +0000]
Now creating the logs about filter manipulation into file so that
systems behaviour can be observed without having too much of verbose
output

5 years agostack-e10k: flows per application argument
Kornilios Kourtis [Wed, 28 Jan 2015 16:38:39 +0000]
stack-e10k: flows per application argument

5 years agosearch: jump now removes flows
Kornilios Kourtis [Wed, 28 Jan 2015 16:19:26 +0000]
search: jump now removes flows

... rather than treating [cc] as a stack.

5 years agosearch: better oracle jump
Kornilios Kourtis [Wed, 28 Jan 2015 13:41:56 +0000]
search: better oracle jump

5 years agoIntegrated dpdk implementation with stack-e10k
Pravin Shinde [Wed, 28 Jan 2015 12:54:11 +0000]
Integrated dpdk implementation with stack-e10k
Added testcase for incremental search issue 'test_incr_pravin_testcase'

5 years agostack-e10k: less verbose help
Kornilios Kourtis [Tue, 27 Jan 2015 19:26:01 +0000]
stack-e10k: less verbose help

5 years agostack-e10k: dpdk backend
Kornilios Kourtis [Tue, 27 Jan 2015 19:22:31 +0000]
stack-e10k: dpdk backend

5 years agoUpdated setIPaddress script to reflect new reality in barrelfish rack.
Pravin [Tue, 27 Jan 2015 17:44:02 +0000]
Updated setIPaddress script to reflect new reality in barrelfish rack.
Pushed some minor install script changes

5 years agocommiting benchmarking script changes. Nothing significant except
Pravin [Tue, 27 Jan 2015 17:37:35 +0000]
commiting benchmarking script changes.  Nothing significant except
reviving linux-onload setup

5 years agoDynamic flow detection and insertion is working.
Pravin Shinde [Tue, 27 Jan 2015 16:04:53 +0000]
Dynamic flow detection and insertion is working.
Created seprate Oracle for Solarflare NIC instead of reusing Intel
oracle

5 years agoincremental search: use it in Stack/stack-e10k
Kornilios Kourtis [Tue, 27 Jan 2015 12:52:44 +0000]
incremental search: use it in Stack/stack-e10k

Done a minimal test with testflows, which seems to work (did not verify
if computations are correct).

5 years agouse new FlowsST interface in stack (non-incremental version)
Kornilios Kourtis [Tue, 27 Jan 2015 09:42:29 +0000]
use new FlowsST interface in stack (non-incremental version)

5 years agoincremental search: some tricks
Kornilios Kourtis [Tue, 27 Jan 2015 09:12:57 +0000]
incremental search: some tricks

5 years agoincremental search: checkpoint
Kornilios Kourtis [Mon, 26 Jan 2015 13:16:56 +0000]
incremental search: checkpoint

5 years agoincremental search: checkpoint
Kornilios Kourtis [Sun, 25 Jan 2015 22:14:51 +0000]
incremental search: checkpoint

5 years agoincremental search: checkpoint
Kornilios Kourtis [Tue, 20 Jan 2015 22:48:08 +0000]
incremental search: checkpoint

5 years agostack: remove ssPrevEndpoints
Kornilios Kourtis [Mon, 19 Jan 2015 17:08:32 +0000]
stack: remove ssPrevEndpoints

This was intended to be used as a way to figure out what changed in the
state when we were using endpoints for the state of stack. Using
ssPrevEndpoints, we could find the added/removed endpoints and implement
an incremental search.

With recent changes, we do not compute the cost function flows from
endpoints. Instead, the application registers them explicitly. Hence,
ssPrevEndpoints is no longer needed.

5 years agosimple ycm configuration file
Kornilios Kourtis [Mon, 19 Jan 2015 17:05:36 +0000]
simple ycm configuration file

5 years agotestflows: simple program that registers flows
Kornilios Kourtis [Mon, 19 Jan 2015 17:05:15 +0000]
testflows: simple program that registers flows

5 years agosearch: error if we do not get any confs from oracle
Kornilios Kourtis [Mon, 19 Jan 2015 17:04:26 +0000]
search: error if we do not get any confs from oracle

5 years agoshmchan: commented out bogus sync() calls
Kornilios Kourtis [Mon, 19 Jan 2015 15:21:26 +0000]
shmchan: commented out bogus sync() calls

5 years agoNow memcached is able to detect new flows and report them to
Pravin Shinde [Fri, 16 Jan 2015 02:58:46 +0000]
Now memcached is able to detect new flows and report them to
dragonet.  Also, made sure that SF driver is working

5 years agoAlso updated memcached to work with new interface
Pravin Shinde [Wed, 17 Dec 2014 17:50:18 +0000]
Also updated memcached to work with new interface
Merge branch 'newflowsapi' into hardcodedSetup

This commit was missing, so added now.

Conflicts:
Dragonet/custom_fancyEcho_run.sh
Dragonet/scripts/pravin/deployPrepare.sh

Conflicts:
Dragonet/apps/microbenchmark/fancyecho.c
Dragonet/c_impl/dpdkDrv/filterMgmt.c
Dragonet/custom_fancyEcho_run.sh
Dragonet/stack/Runner/dynamic.c
benchmarking/memcached/dnet_interface.c

5 years agoStack: remove unused trigger
Kornilios Kourtis [Thu, 15 Jan 2015 17:32:11 +0000]
Stack: remove unused trigger

5 years agoupdated benchmarking scripts for running load-balancing setup with
Pravin [Thu, 15 Jan 2015 13:00:36 +0000]
updated benchmarking scripts for running load-balancing setup with
hardcoded setup.

5 years agoupdated stack-sf code which was lagging behind, added support for
Pravin Shinde [Thu, 15 Jan 2015 12:55:52 +0000]
updated stack-sf code which was lagging behind, added support for
hardcoded balancing scenario, changed the IP addresses, MACs to
babybel2, updated wrapper sripts to detect stack starting more
reliably.

Conflicts:
Dragonet/Scenarios/S3.hs

5 years agochanges in benchmarking harness scripts to support new setup, including
Pravin [Mon, 12 Jan 2015 13:04:26 +0000]
changes in benchmarking harness scripts to support new setup, including
hardcoded oracle

5 years agoadded plotting script which works with newer format, and also
Pravin Shinde [Mon, 12 Jan 2015 13:01:38 +0000]
added plotting script which works with newer format, and also
does not depend on hardcoded paths

5 years agoFixed the issues with starting 1000 flows (atleast with hardcoded oracle)
Pravin Shinde [Mon, 12 Jan 2015 02:19:41 +0000]
Fixed the issues with starting 1000 flows (atleast with hardcoded oracle)

Conflicts:
Dragonet/stack/stack-dpdk.hs
benchmarking/memcached/dnet_interface.c

5 years agoFixed the issue of missing sockets for few pipelines/partitions when
Pravin Shinde [Wed, 24 Dec 2014 18:34:55 +0000]
Fixed the issue of missing sockets for few pipelines/partitions when
there are more queues/partitions than spanned sockets

5 years agoFixed issue of a race condition in bulk-transport
Pravin Shinde [Thu, 18 Dec 2014 17:11:06 +0000]
Fixed issue of a race condition in bulk-transport

which was causing a assert failure while binding a channel which is not
initialized.

Conflicts:
Dragonet/custom_fancyEcho_run.sh

5 years agohardcoded trigger is working for stack-dpdk now.
Pravin Shinde [Mon, 15 Dec 2014 10:16:43 +0000]
hardcoded trigger is working for stack-dpdk now.

Also improves the scripts to activate dpdk driver

Conflicts:
Dragonet/c_impl/impl_dpdk.c
Dragonet/custom_fancyEcho_run.sh
Dragonet/scripts/pravin/deployPrepare.sh
Dragonet/stack/Stack.hs
Dragonet/stack/stack-dpdk.hs

5 years agoAdded code to store all flows before calling updateFlows.
Pravin Shinde [Mon, 8 Dec 2014 22:49:00 +0000]
Added code to store all flows before calling updateFlows.

This still needs working as I am getting following error when I run:
Warning: Task starting at no longer existing node, dropping task...

Conflicts:
Dragonet/custom_fancyEcho_run.sh
Dragonet/lib/Dragonet/Pipelines/Dynamic.hs
Dragonet/stack/Stack.hs

5 years agoCreating and returning the whole hardcoded configuration once in
Pravin Shinde [Mon, 8 Dec 2014 15:17:54 +0000]
Creating and returning the whole hardcoded configuration once in
the begining, instead of returning one flow at time.

5 years agoincremental search: use a ST.ST flowcache
Kornilios Kourtis [Wed, 14 Jan 2015 12:23:44 +0000]
incremental search: use a ST.ST flowcache

Still ugly, but now with flowcache

Results Summary:
  Flows        Search (HF)  IncrSearch     IncrSearch (HF)
   10            9.2ms        27.5ms          25.34ms
   100           1.2s          1.6s           929ms
   200           9.1s          5.7s           3.5s
   500         105.2s         24.0s           18s

A bit better but still not great...

5 years agoincremental search: first benchmark
Kornilios Kourtis [Wed, 14 Jan 2015 09:26:08 +0000]
incremental search: first benchmark

This is ugly as sin currently, but it allows for initial benchmarking,
which shows that the scalability is better for the incremental case.

Results:
benchmarking search: 10
mean: 9.263010 ms, lb 9.241076 ms, ub 9.277315 ms, ci 0.950
benchmarking search: 100
mean: 1.253582 s, lb 1.204199 s, ub 1.280288 s, ci 0.950
benchmarking search: 200
mean: 9.102355 s, lb 9.073130 s, ub 9.126548 s, ci 0.950
benchmarking search: 500
mean: 105.1918 s, lb 104.8949 s, ub 105.4493 s, ci 0.950
benchmarking incremental search: 10
mean: 28.10428 ms, lb 27.57093 ms, ub 28.46159 ms, ci 0.950
benchmarking incremental search: 100
mean: 1.636281 s, lb 1.627021 s, ub 1.645387 s, ci 0.950
benchmarking incremental search: 200
mean: 5.731899 s, lb 5.723379 s, ub 5.740216 s, ci 0.950
benchmarking incremental search: 500
mean: 24.05272 s, lb 24.03319 s, ub 24.06688 s, ci 0.950

Summary:
 Flows        Search          IncrSearch
  10            9.2ms            27.5ms
  100           1.2s             1.6s
  200           9.1s             5.7s
  500         105.2s            24.0s

5 years agoincrementalSearch: simple incremental config test works
Kornilios Kourtis [Tue, 13 Jan 2015 17:57:41 +0000]
incrementalSearch: simple incremental config test works

5 years agomove is{R,T}xQueueNode to PGU
Kornilios Kourtis [Tue, 13 Jan 2015 08:38:45 +0000]
move is{R,T}xQueueNode to PGU

5 years agomove QueueId to Dragonet.Conventions
Kornilios Kourtis [Tue, 13 Jan 2015 08:37:06 +0000]
move QueueId to Dragonet.Conventions

5 years agoflowmap updates
Kornilios Kourtis [Mon, 12 Jan 2015 12:58:02 +0000]
flowmap updates