Channel (Channel)

Introduction to Golang

Golang is one of the currently popular languages, with native support for concurrency, convenient thread operations, lightweight coroutines, and the maximum utilization of multi-core CPUs. Developing port scanners with it not only makes the development process efficient but also ensures excellent program performance. Moreover, Go supports cross-platform compilation and single-platform cross-compilation, making it extremely convenient to use.

The concurrency of Go language is based on goroutines, which are similar to threads but not threads. Goroutines can be understood as a type of virtual thread. The Go runtime participates in scheduling goroutines and allocates them reasonably to each CPU, maximizing CPU performance.

In multiple goroutines, Go language uses channels for communication. Channels are a built-in data structure that allows users to synchronize sending typed messages between different goroutines. This makes the programming model tend to send messages between goroutines rather than have multiple goroutines compete for the use of the same data.

These advantages are very suitable for writing scanners.

Multi-threading

1.Understanding concurrency and parallelism

Correcting a common misunderstanding of most people about multi-concurrency, concurrency is not parallelism. For example, for network connection requests, if there are 1000 network connections to be processed at present, concurrency is to connect one by one. After sending out the current connection, it will suspend the task to initiate a connection to the next target while waiting for the return connection. After the connection of this task returns, it will continue to process. By switching between multiple threads, the purpose of reducing physical processor idle waiting time is achieved. A typical language is Python, whose multi-threading is concurrency and does not utilize parallelism. This kind of concurrency can improve speed to some extent in processing network connections and IO operations, but there is basically no improvement in the pure CPU calculation scenario. Parallelism is relatively easy to understand, which is to use the multiple cores of the CPU to run multiple tasks. Originally, a task that takes 10 minutes to run in parallel can be completed in 5 minutes with two threads (this can be understood in this way, but in fact it is not 5 minutes).

The simplest example of concurrency is running multiple goroutines on a single logical processor, which can be simply said to be running multiple goroutines on a single CPU core, for example, an Intel with four cores generally has 8 threads, concurrency can only utilize one, while parallelism can utilize multiple ones. In Go, runtime.GOMAXPROCS(2) is used to specify the number of threads to be used.

01Concurrency

Logical processor: The runtime of Golang schedules goroutines to run on logical processors. Each logical processor is bound to an operating system thread. In versions of Golang 1.5 and later, the runtime will default to allocating one logical processor for each available physical processor.

Local run queue: Each logical processor has a local run queue. If a goroutine is created and ready to run, it will first be placed in the scheduler's global run queue. After that, the scheduler will allocate a goroutine from the global run queue to a logical processor and place it in the local run queue of this logical processor. The goroutines in the local run queue will wait until the assigned logical processor executes them.

This is concurrency, with only one logical processor that can execute only one task at a time.

These two tasks are executed in a round-robin manner, and it will try to minimize CPU idle time to improve speed.

02Parallel

Parallelism can significantly improve speed, but the corresponding CPU resource consumption will also increase.

This is a very spiritual picture to illustrate the distinction between the two.

Experience concurrency by yourself

func main() {import ("fmt""runtime""sync""time")func main() {// wg is used to wait for the program to completestime:=time.Now()defer func() {fmt.Println(time.Since(stime))}()var wg sync.WaitGroup// Allocate one logical processor for the scheduler to useruntime.GOMAXPROCS(2)  // Limit the number of processors useda:=runtime.NumCPU()println(a)// Increment the count by 2, indicating that two goroutines need to be waited forwg.Add(2)// Create two goroutinesfmt.Println("Create Goroutines")go printPrime("A", &wg)go printPrime("B", &wg)// Wait for the goroutine to endfmt.Println("Waiting To Finish")wg.Wait()fmt.Println("Terminating Program")// printPrime displays prime numbers less than 5000func printPrime(prefix string, wg *sync.WaitGroup){// Call Done when the function exits to notify the main function that the work has been completeddefer wg.Done()next:for outer := 2; outer < 100000; outer++ {for inner := 2; inner < outer; inner++ {if outer % inner == 0 {continue nextfmt.Printf("%s:%d\n", prefix, outer)fmt.Println("Completed", prefix)

Coroutine and Thread

01Coroutine

Independent stack space, shared heap space, scheduling is controlled by the user, and essentially it is somewhat similar to user-level threads, and the scheduling of these user-level threads is also implemented by themselves.

02Thread

Multiple coroutines can run on a single thread, and coroutines are lightweight threads.

goroutine

Goroutine is a lightweight thread implementation in Go language, managed by the Go runtime (runtime). The Go program intelligently allocates tasks in the goroutine to each CPU. Detailed introduction of goroutine

A Go program starts from the main() function in the main package, and a default goroutine is created for the main() function when the program starts.

It is very convenient to use

func main() {go say("word")say("hello")func say(s string)  {for i:=0;i<5;i++{time.Sleep(time.Second)fmt.Println(s)

The only thing to note is that the example above can output the result we want normally.

word

hello
hello
word
word
hello
hello
word
hello

But if you remove say("hello") it won't work, the output will become empty because the goroutine will exit with the main thread, so some means need to be used to control the exit time of the goroutine reasonably, and block the main thread when necessary (most of the time it needs to) to prevent the main thread from exiting and causing the sub-thread to be interrupted unexpectedly.

Here are three examples

The first method uses the sleep function:

func main() {go say("word")time.Sleep(time.Second*6)func say(s string)  {for i:=0;i<5;i++{time.Sleep(time.Second)fmt.Println(s)

It can be, but do you feel it's a bit silly?

The second method uses channel blocking:

func main() {c:=make(chan bool)go say("word",c)<-cfunc say(s string,c chan bool)  {for i:=0;i<5;i++{time.Sleep(time.Second)fmt.Println(s)c<-true

Use the inherent characteristics of the channel to block the main thread, and the program will continue to run downward only when the goroutine has executed and written data to c, and the main thread has read the data in c.

The third and most commonly used method is to solve it with WaitGroup in the sync package

func main() {var wg sync.WaitGroupwg.Add(1)go say("word",&wg)wg.Wait()func say(s string,wg *sync.WaitGroup)  {defer wg.Done()for i:=0;i<5;i++{time.Sleep(time.Second)fmt.Println(s)

It should be noted that add must be in the same thread as wait, otherwise the main thread may end before the goroutine is added to the wg.

Channel (Channel)

Declaration

var ch chan intvar ch map[string] chan boolch := make(chan int)

A channel is a communication mechanism that allows a goroutine to send value information to another goroutine through it. Each channel has a special type, which is the type of data that can be sent through the channels.

Go语言提倡使用通信的方法代替共享内存，当一个资源需要在 goroutine 之间共享时，通道在 goroutine 之间架起了一个管道，并提供了确保同步交换数据的机制。声明通道时，需要指定将要被共享的数据的类型。可以通过通道共享内置类型、命名类型、结构类型和引用类型的值或者指针。

At any time, only one goroutine can access the channel to send and receive data. Goroutines communicate through channels, which act like a conveyor belt or queue and always follow the first-in-first-out (queue) rule to ensure the order of data transmission. The Go language advocates using communication methods instead of shared memory. When a resource needs to be shared between goroutines, a channel establishes a pipeline between goroutines and provides a mechanism to ensure the synchronization of data exchange. When declaring a channel, it is necessary to specify the type of data that will be shared. Values or pointers of built-in types, named types, struct types, and reference types can be shared through channels.

func main() {s := []int{7, 5, 3, 7, 4, 3}c := make(chan int)go sum(s[:len(s)/2], c)go sum(s[len(s)/2:], c)x := <-cy := <-cfmt.Println(x, y, x+y)func sum(s []int, c chan int) {sum := 0for _, v := range s {sum += vc <- sum

To speak of (unbuffered) channels alone, it is just a channel that does not store anything. It is just a pipeline that connects two goroutines and transmits specified types. Data will only be transmitted when both parties are ready, that is, when they are connected to this pipeline at the same time. If only one party connects to the pipeline, it will not put data into the pipeline and do other things, but will wait for the other end of the pipeline to connect. During this waiting period, the goroutine will remain in a blocked state, which is why channel control is used for the main thread above

To sum up,

01 Writing data to the channel usually causes the program to hang until another goroutine reads data from this channel

02 If there is no data in the channel, reading data from the channel will also cause the program to block until data is written into the channel

If there is only such a pipeline that can only transmit data, it is obviously not capable of most situations. There is a demand, and there will be solutions. The buffered channel comes into being

A buffered channel is a channel that can store one or more values before being received. This type does not require that the sender and receiver must complete the send and receive operations at the same time. The blocking conditions have also changed (blocking only occurs when the buffer is full, blocking occurs when there is nothing in the buffer but there is a receiving action, and unbuffered channels are constantly blocking)

 Compare the two in the figure above

After seeing the buffered channel, isn't the producer-consumer model immediately popping into your mind? The scanner will fully utilize the advantages of the buffered channel to save a large amount of memory space

The declaration method is also very simple

ch := make(chan int, 10)

However, with buffered channels, there is a need to involve closing issues. In order to prevent problems (writing data to a closed channel), it is necessary to strictly follow the rules, only the sender can close the channel, and the receiver should not close the channel.

Summary

Combined with the lightweight switching of goroutines, it is convenient to switch threads in user mode, has small creation and destruction costs, can maximize the use of multi-core by nature, and has the advantage of channels, which are very suitable for network scanning development.

Network communication

1TCP connection

The standard library of go is very mature, and many things can be implemented through the standard library. Network communication is basically in the net library.

Here, the most commonly used network connection functions are introduced.

Dial

func Dial(network, address string) (Conn, error)

It only needs two parameters and returns a network connection and an error

TCP connection: conn, err := net.Dial("tcp", "192.168.10.10:80")UDP connection: conn, err := net.Dial("udp", "192.168.10.10:8888")ICMP connection: conn, err := net.Dial("ip4:icmp", "c.biancheng.net")ICMP connection: conn, err := net.Dial("ip4:1", "10.0.0.3")

It can be used to detect whether the port is open

func main() {conn, err := net.Dial("tcp", "127.0.0.1:80")if err!=nil{fmt.Println(err)returnconn.Close()fmt.Println("Port open")

If tcp can connect, it proves that the port is open

DialTimeout

func DialTimeout(network, address string, timeout time.Duration) (Conn, error)

The usage method of DialTCP is the same as that of Dial, but it adds a timeout, which is very commonly used because it prevents blocking during the scanning process.

DialTCP

func DialTCP(network string, laddr, raddr *TCPAddr) (*TCPConn, error)

Similar to the Dial usage method, it adds an laddr that can specify the address used by the connection, if it is empty it will randomly select, raddr is the target address we want to connect to, which is a TCPAddr type and can be obtained using ResolveTCPAddr

func ResolveTCPAddr(network, address string) (*TCPAddr, error)

Dialer struct

type Dialer struct {Timeout time.DurationDeadline time.TimeLocalAddr AddrDualStack boolFallbackDelay time.DurationKeepAlive time.DurationResolver *ResolverCancel <-chan struct{}Control func(network, address string, c syscall.RawConn) error

You can create a customizable connection

func (d *Dialer) Dial(network, address string) (Conn, error)

You can also create a connection with context control

func (d *Dialer) DialContext(ctx context.Context, network, address string) (Conn, error)

2ICMP

Use the golang.org/x/net/icmp library

ListenPacket

Use ListenPacket to establish a listener, which can receive icmp packets sent by other hosts

Send icmp packets using the WriteTo method require specifying the target address

func (c *PacketConn) WriteTo(b []byte, dst net.Addr) (int, error)

dst, _ := net.ResolveIPAddr("ip", 目标地址)IcmpByte := makemsg(host.String())    // Construct the packetconn.WriteTo(IcmpByte, dst)    // Send the packet to the destination

Use Readfrom to determine who sent the return packet

func (c *PacketConn) ReadFrom(b []byte) (int, net.Addr, error)

msg := make([]byte, 100)_, sourceIP, _ := conn.ReadFrom(msg)

Obtaining an IP address indicates that the host of that IP address is alive

I/O Operations

Once the network connection is established, we can know that the target is alive. Next, if we need to determine what the port is doing, we need io operations, or if we need to save the scan results locally, we also need io operations

In the io standard library, two important interfaces are defined, Reader and Writer, and all io operations are basically centered around these two interfaces

And these two interfaces are implemented in many standard libraries in Go, such as the io library, ioutil library, bufio library, bytes library, strings library, and so on.

For example, after establishing a TCP connection, we get a net.tcpconn object, which implements the Writer and Reader interfaces, allowing us to write to the connection and also read from it

A simple organization, not comprehensive. There are many ways to use and convert, which can be explored independently

Summary

After having established these three important foundations, writing the scanner becomes very simple.