Proxmox and replacing disk in ZFS pool

For one time this blog post will not be around Telecom and Cisco/Juniper/Nokia or something like this.

Just to keep in mind how to replace a faulty device in a ZFS pool.

I have :

root@pve:~# zpool status -x
root@pve:~#  zpool status
  pool: pve-zfs
 state: DEGRADED
status: One or more devices could not be used because the label is missing or
        invalid.  Sufficient replicas exist for the pool to continue
        functioning in a degraded state.
action: Replace the device using 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-4J
  scan: resilvered 41.9M in 0 days 00:00:11 with 0 errors on Sun Jul 24 13:38:51 2022
config:

        NAME                        STATE     READ WRITE CKSUM
        pve-zfs                     DEGRADED     0     0     0
          mirror-0                  DEGRADED     0     0     0
            wwn-0x50014ee267b78b52  ONLINE       0     0     0
            2534239155907356895     FAULTED      0     0     0  was /dev/sdb1
          mirror-1                  ONLINE       0     0     0
            wwn-0x50014ee267b63342  ONLINE       0     0     0
            wwn-0x50014ee2bd0cf6b4  ONLINE       0     0     0

errors: No known data errors

But how to replace this faulty device when all the howto on the net talk about replace/make offline the old disk… But in my situation I have made an RMA on the disk and don’t have mind to make the faulty device offline.

Nevertheless, I have replaced my 2TB disk with a new one, such as :
But If I made :

root@pve:~# zpool replace pve-zfs   2534239155907356895  ata-WDC_WD20EFRX-68EUZN0_WD-WCC4M1EUJ8KN
invalid vdev specification
use '-f' to override the following errors:
/dev/disk/by-id/ata-WDC_WD20EFRX-68EUZN0_WD-WCC4M1EUJ8KN-part1 contains a filesystem of type 'ntfs'

After make a little apt-get install parted :

root@pve:~# parted /dev/sda
GNU Parted 3.2
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print                                                            
Model: ATA WDC WD20EFRX-68E (scsi)
Disk /dev/sda: 2000GB
Sector size (logical/physical): 512B/4096B
Partition Table: msdos
Disk Flags: 

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  2000GB  2000GB  primary  ntfs

(parted) rm 1                                                             
(parted) print                                                            
Model: ATA WDC WD20EFRX-68E (scsi)
Disk /dev/sda: 2000GB
Sector size (logical/physical): 512B/4096B
Partition Table: msdos
Disk Flags: 

Number  Start  End  Size  Type  File system  Flags

(parted) mklabel GPT                                                      
Warning: The existing disk label on /dev/sda will be destroyed and all data on this disk will be lost. Do you want to continue?
Yes/No? Yes                                                               
(parted) q                                                                
Information: You may need to update /etc/fstab.

root@pve:~#

So :

root@pve:~# zpool replace pve-zfs   2534239155907356895  ata-WDC_WD20EFRX-68EUZN0_WD-WCC4M1EUJ8KN
root@pve:~# zpool status -x
  pool: pve-zfs
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sun Jul 24 14:23:11 2022
        10.5G scanned at 716M/s, 4.04G issued at 276M/s, 450G total
        0B resilvered, 0.90% done, 0 days 00:27:37 to go
config:

        NAME                                            STATE     READ WRITE CKSUM
        pve-zfs                                         DEGRADED     0     0     0
          mirror-0                                      DEGRADED     0     0     0
            wwn-0x50014ee267b78b52                      ONLINE       0     0     0
            replacing-1                                 DEGRADED     0     0     0
              2534239155907356895                       FAULTED      0     0     0  was /dev/sdb1
              ata-WDC_WD20EFRX-68EUZN0_WD-WCC4M1EUJ8KN  ONLINE       0     0     0
          mirror-1                                      ONLINE       0     0     0
            wwn-0x50014ee267b63342                      ONLINE       0     0     0
            wwn-0x50014ee2bd0cf6b4                      ONLINE       0     0     0

errors: No known data errors
root@pve:~#

How I get the new device name :

root@pve:~# ls -l /dev/disk/by-id | grep J8KN
lrwxrwxrwx 1 root root  9 Jul 24 14:23 ata-WDC_WD20EFRX-68EUZN0_WD-WCC4M1EUJ8KN -> ../../sda
lrwxrwxrwx 1 root root 10 Jul 24 14:23 ata-WDC_WD20EFRX-68EUZN0_WD-WCC4M1EUJ8KN-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Jul 24 14:23 ata-WDC_WD20EFRX-68EUZN0_WD-WCC4M1EUJ8KN-part9 -> ../../sda9
root@pve:~# 

Where “J8KN” is a pattern of the Serial Number you can pick on the new disk.

Bind9 as master DNS server, NSD as backup with TSIG.

Perhaps it seems easy for you, but it was not for me at begining even if I am running bind9 master/bind9 slave already.

We will begin with TSIG key, on your master DNS server :

root@Sagitarius:~# tsig-keygen mykey > /etc/bind/keys/mykey.key
root@Sagitarius:~# cat /etc/bind/keys/mykey.key
key "mykey" {
        algorithm hmac-sha256;
        secret "+OyXk+FYgsnE6Lei59Qi2LPsctRHPaqcYRpftG4YXKk=";
};
root@Sagitarius:~# 

Now we will configure this on my “named.conf.local” file as :

root@Sagitarius:/etc/bind# cat named.conf.local 
[...]
//On indique la clé à utiliser et son algorithme de chiffrement
include "/etc/bind/keys/mykey.key";

//On indique l'adresse IP du serveur "Esclave"
server 192.168.1.81
{
        keys { mykey; };
};
[...]
zone "my-blah-zone.fr" {
       type master;
       file "/etc/bind/my-blah-zone.fr.zone";
       allow-transfer { 192.168.1.81; key mykey; };
       notify yes;
};
[...]

Ok. last command :

root@Sagitarius:~# rndc reload
root@Sagitarius:~#

Ok good, now let’s see how it is configured on NSD host.
It’s really easy :

clucas@slave:/etc/nsd$ cat /etc/nsd/nsd.conf.d/secondaries/my-blah-zone.fr.conf 
key:
  name: "mykey"
  algorithm: hmac-sha256
  secret: "+OyXk+FYgsnE6Lei59Qi2LPsctRHPaqcYRpftG4YXKk="


zone:
        # this server is secondary,  is master.
        name: my-blah-zone.fr
        allow-notify: 192.168.1.43 mykey
        request-xfr:  192.168.1.43 mykey

Now :

nsd-checkconf /etc/nsd/nsd.conf
nsd-control reload
nsd-control status

It seems easy, but I have take long time to see that all these items bellow must be identical :

  • name of the key
  • algorithm
  • secret

Easy as hell !

PS : All this can be controlled by Ansible, but it is another story.

NextCloud and Memcached-APCu

For those of you which update your personal cloud to the lastest stable version of Nextcloud due to CVE (https://www.cert.ssi.gouv.fr/avis/CERTFR-2021-AVI-543/), you could encountered this error :

An unhandled exception has been thrown:
OC\HintException: [0]: Memcache \OC\Memcache\APCu not available for local cache (Is the matching PHP module installed and enabled?)

You could solve the issue :

echo 'apc.enable_cli=1' >> /etc/php/7.x/mods-available/apcu.ini

Have fun.

bwping patch (catching signals)

diff -urpN bwping/bwping.c bwping-patched/bwping.c
--- bwping/bwping.c     2012-10-11 19:23:17.000000000 +0200
+++ bwping-patched/bwping.c     2017-04-20 09:06:23.449540033 +0200
@@ -26,6 +26,7 @@
 #include 
 #include 
 #include 
+#include 
 
 #ifdef __CYGWIN__
 #include "cygwin.h"
@@ -224,21 +225,39 @@ static int recv_ping (int sock, int iden
         return 0;
     }
 }
+unsigned int   transmitted_number, received_number;
+unsigned long  int received_volume;
+struct timeval begin, end;
+
+void sig_handler(int signo)
+{
+       if (signo == SIGUSR1) {
+                printf("Total: pkts sent/rcvd: %u/%u, volume rcvd: %lu bytes, time: %d sec, speed: %lu kbps, rtt min/max/average: %llu/%llu/%llu ms\n",
+                               transmitted_number, received_number, received_volume, (int)(end.tv_sec - begin.tv_sec),
+                               end.tv_sec - begin.tv_sec?((received_volume / (end.tv_sec - begin.tv_sec)) * 8) / 1000:(received_volume * 8) / 1000,
+                               min_rtt==DEF_MIN_RTT?0:min_rtt, max_rtt, average_rtt);
+               exit(255);
+       }
+}
 
 int main (int argc, char **argv)
 {
     int                    sock, exitval, ch, ident, finish, pktburst, i, n;
-    unsigned int           bufsize, tos, transmitted_number, received_number;
-    unsigned long int      kbps, pktsize, volume, rperiod, received_volume;
+    unsigned int           bufsize, tos;
+    unsigned long int      kbps, pktsize, volume, rperiod;
     unsigned long long int min_interval, interval, current_interval, integral_error;
     char                   *ep, *bind_addr, *target;
     fd_set                 fds;
     struct sockaddr_in     bind_to, to;
     struct hostent         *hp;
-    struct timeval         begin, end, report, start, now, seltimeout;
+    struct timeval         report, start, now, seltimeout;
 
     sock = socket(AF_INET, SOCK_RAW, IPPROTO_ICMP);
 
+       if (signal(SIGUSR1, sig_handler) == SIG_ERR)
+               printf("\ncan't catch SIGUSR1\n");
+
+
     if (sock==-1) {
         perror("bwping: socket(AF_INET, SOCK_RAW, IPPROTO_ICMP) failed");

Download : patch-bwping-sig.diff

jquery – jqgrid : Custom delete handler

It may possible you are face off the problem to synchronize some other items in your GUI when you delete a row in your jqgrid.

You can use this tip to do it :

var myDelOptions = {
    onclickSubmit: function(options, rowid) {
          var grid_id = $.jgrid.jqID($( "#tip" )[0].id),
             grid_p = $( "#tip" )[0].p,
             newPage = grid_p.page;

          // reset the value of processing option which could be modified
          options.processing = true;

          // delete the row
          $( "#tip" ).delRowData(rowid);
          $.ajax({
                url: 'backend/ip.php',
                type: 'POST',
                data : 'oper=del&id=' + rowid,
                dataType: "text",
                success: function(data, status, xr) {
                         $( "#troute" ).trigger("reloadGrid");
                         $( "#tip" ).trigger("reloadGrid");
                },
                error: function(e) {
                         //called when there is an error
                         //console.log(e.message);
                }
           });
           
           $.jgrid.hideModal("#delmod"+grid_id,
                             {gb:"#gbox_"+grid_id,
                              jqm:options.jqModal,onClose:options.onClose});

           if (grid_p.lastpage > 1) {// on the multipage grid reload the grid
                    if (grid_p.reccount === 0 && newPage === grid_p.lastpage) {
                        // if after deliting there are no rows on the current page
                        // which is the last page of the grid
                        newPage--; // go to the previous page
                     }
                     // reload grid to make the row from the next page visable.
                     $( "#tip" ).trigger("reloadGrid", [{page:newPage}]);
            }

            return true;
     },
     processing:true
}; // fin variable

 $("#tip").jqGrid({
     url: "backend/ip.php",
     datatype: "xml",
     mtype: "GET",
     colNames: [" ", "Prefix", "Longueur", "Device"],
     colModel: [
                { name: 'myac', width:80, fixed:true, sortable:false, resize:false,
                  formatter:'actions', formatoptions:{editbutton: false, onedit:null,
                  delbutton:true, delOptions: myDelOptions}},
                { name: "prefix", width: 125, align: "center" },
                { name: "longueur", width: 100, align: "center" },
                { name: "device", width: 75, align: "center" }
      ],
      [...]
      caption: "Affectation IP / Intf"
});

[...]

The great thing is that you are able to make some data / gui treatment. Here I make a data update by means of a jquery ajax call and update jqgrids by means of two trigger(“reloadGrid”) calls.

Hope this can help

Etherate : the beginning of an Open Aurora Tango test (L2 tester)…

For those of you which perform some Aurora Tango tests (BERT, RFC 2544) you may know that a couple of these testers are really expensive ! The principle of these test is to place a tester in loopback and another is test mode. By means of this, you are able to qualify an Ethernet link (direct, Q-in-Q, VPLS, Xconnect).

During my readings of mailing-lists such as cisco-nsp or others, someone posts an URL to a really awesome Linux program : etherate. With modern CPU platform you are able to supply a throughput until 1G, and this program give you the possibility to test your L2 circuits.

You can do some tests by using it such as :

 

root@Loop_Host# etherate -r

And :

root@Tx_Host# etherate

To give some features :

root@pluton:~/COMPILE/Etherate-master# ./etherate -h
Usage info; [Mode] [Destination] [Source] [Options] [Other]
[Mode] By default run in transmit mode, not receive
    -r    Change to receive (listening) mode.
[Destination]
    -d    Without this we default to 00:00:5E:00:00:02
        as the TX host and :01 as the RX host.
        Specify a custom desctination MAC address, 
        -d 11:22:33:44:55:66
[Source]
        Specify a custom source MAC address, -s 11:22:33:44:55:66
    -i    Set interface by name. Without this option we guess which
        interface to use.
    -I    Set interface by index. Without this option we guess which
        interface to use.
    -l    List interface indexes (then quit) for use with -i option.
    -s    Without this we default to 00:00:5E:00:00:01
        as the TX host and :02 as the RX host.
[Options]
    -a    Ack mode, have the receiver ack each frame during the test
        (This will significantly reduce the speed of the test).
    -b    Number of bytes to send, default is 0, default behaviour
        is to wait for duration.
        Only one of -t, -c or -b can be used, both override -t,
        -b overrides -c.
    -c    Number of frames to send, default is 0, default behaviour
        is to wait for duration.
    -e    Set a custom ethertype value the default is 0x0800 (IPv4).
    -f    Frame payload size in bytes, default is 1500
        (1514 bytes is the expected size on the wire with headers).
    -m    Max bytes per/second to send, -m 125000 (1Mbps).
    -t    Transmition duration, integer in seconds, default is 30.
[Other]
    -v    Add an 802.1q VLAN tag. By default none is in the header.
        If using a PCP value with -p a default VLAN of 0 is added.
    -p    Add an 802.1p PCP value from 1 to 7 using options -p 1 to
        -p 7. If more than one value is given, the highest is used.
        Default is 0 if none specified.
        (If no 802.1q tag is set the VLAN 0 will be used).
    -q    Add an outter Q-in-Q tag. If used without -v, 1 is used
        for the inner VLAN ID.
        #NOT IMPLEMENTED YET#
    -o    Add an 802.1p PCP value to the outer Q-in-Q VLAN tag.
        If no PCP value is specified and a Q-in-Q VLAN ID is,
        0 will be used. If no outer Q-in-Q VLAN ID is supplied this
        option is ignored. -o 1 to -o 7 like the -p option above.
        #NOT IMPLEMENTED YET#
    -x    Display examples.
        #NOT IMPLEMENTED YET#
    -V|--version Display version
    -h|--help Display this help text
root@pluton:~/COMPILE/Etherate-master#

I really hope and try to help so that this project goes to his end. It could be really great to have L2 tests as this one as we have with bwping and iperf on L3/L4 layer of OSI model.

I am disappointed by Ubiquiti Networks

It is been a while I check forums and Ubiquiti products. They are really good products. At the beginning the really interesting thing in ubiquiti was the Open minded / Open Source direction they used. I begun to use their product with SR and XR card (when they don’t seek any final products). It was really easy to make your own product firmware with an embedded MIPS motherboard and their cards. The SDK was available at this time.

I check recently and now it is not. Ubiquiti has been introduce in Wall Street. I am really disappointed because when I was interviewed by them for their papers to be introduce to Wall Street I insisted on the fact of they success OpenSource is one main of it.

I would want to make some patch on kernel to be able to pass PPP/IP DSCP to their Atheros card to be able to do the same thing with IP (http://wiki.ubnt.com/AirMax_-_QoS_DSCP/TOS_Mappings). Now SDK is not available and I am really disappointed.

Linux : Remote desktop && Hamachi

For those of you which need to access by means of “Remote Desktop” you can use :

  • Nomachine NX
  • Teamviewer
  • VNC

And you can use both LogmeIn Hamachi and VNC. Hamachi is a software which can make network betweeen devices. Against logmein product which lot of you know, here it is at ~~Level2~~ (OSI).

 

How you create your VPN ?

For Linux users, you must install vnc4server and hamachi (supplied by LogmeIn) package (dpkg is your friend). Then you launch your vnc server :

root@plop# vncserver 
New 'plop:1 (clucas)' desktop is plop:1

Starting applications specified in /home/clucas/.vnc/xstartup
Log file is /home/clucas/.vnc/plop:1.log

Now the VNC server is reacheable by means of 5901 (5900 + 1) port.

Then you can create your VPN network by using hamachi :

root@neptune:~# hamachi -h
LogMeIn Hamachi, a zero-config virtual private networking utility, ver 2.1.0.76

  usage: hamachi [command]

  command    specifies an action. Can be one of the following -

             #  set-nick 
                login
                logon
                logout
                logoff
             #  list
                peer 
                network 
             #  create  []
                set-pass  []
                set-access  [lock|unlock] [manual|auto]
                delete 
                evict  
             #  approve  
                reject  
             #  join  []
                do-join  []
                leave 
             #  go-online 
                go-offline 
             #  attach 
                attach-net 
                cancel
             #  gw-config 
                    [dhcp|static [net    ]
                                 [domain ]
                                 [dns  []]]
                    [del  ...]
                    [add  ...]
             #  set-ip-mode ipv4 | ipv6 | both
             #  check-update
                vpn-alias  |0
  If no command is specified, hamachi displays its status including version,
  pid, client id, online status, nickname and the LogMeIn account
root@neptune~# hamachi attach clucas@altern.org
root@neptune~# hamachi join network password
root@neptune~# hamachi go-online network

You have now access to your Linux server/desktop from anywhere by means of hamachi’s device :

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

For mac users (such as me : not a really experimented user), you have a built-in VNC client in MAC from 10.5 MAC OS X.

Go in your finder window > Go > Connect to server >
Now you can access to your device by means of :
vnc://x.y.z.w:5901 (where 5900 + X display : here :1)

‘netcat’ : my best friend :)

For those of you who read this unpretentious blog, you must have noticed this week-end a maintenance window.

The DD of my hosting has crashed. To be more precise, it is crashing. His time to live is near to expire…

So to save my files on this server, a simple tar and scp is not enough. In fact, if you do this, you will create inodes and store on your FS. You will have lot of chance to have corrupted tarballs.

So I chose to use netcat to make my saves. tarballs are created on the fly on the network. So you only make reads on your DD, no creates.

On the device you have datas :

source# tar c your_folder | nc -q 10 -l -p 7777

 

On the remote end :

target# nc -w 10 source.domain.net 7777 > your_folder.tar

 

You can use the ‘z’ flag to gzip the archive. For more explaination, man is your friend, or in others words : RTFM…