A [[hypervisor]] OS ## Documentation - [official docs](pve-admin-guide.pdf) ## setup 0. [Great installation video](https://youtu.be/azORbxrItOo) 1. [download iso](https://proxmox.com/en/downloads) 2. Use management interface on the server, [[Integrated Dell Remote Access Controller|iDRAC]] 1. Turn server on 2. Attach virtual media (the .iso) 3. Follow the prompts and fill in the information 1. chose [[RAID#RAID-Z]]1 for a mirrored boot drive on SSD pair 1. if SSD's are not visible on boot, check that the SATA settings in device settings is set to ACHI and auto 4. fill in network information 1. Much of it can be gleaned from your router console 2. `ifconfig get default | grep gateway` for the default gateway 3. for MacOS to find [[DNS]] server just search for [[DNS]] in *system preferences* 5. Once [[Proxmox]] is finished bootstrapping and restarts the server, remove the virtual media 6. if screen loads to a console then you can just transition back to your main machine and use the web interface to finish. 7. management console is at: `https://#.#.#.#:8006` (replace `#` with valid IPV4 address) 8. connection will be insecure and that's okay, proceed anyhow 9. login as `root` with the password you previously set. 10. Ignore subscription popup, its FOSS unless you want enterprise subscription ## Post Install Configuration - [ ] TODO reformat this section by separating out the steps to their own notes and making this list more legible 1. `apt-get install neovim ranger tldr cifs-util` 2. [[Proxmox#Configuration]] ## Configuration ### Amazing Proxmox Repository https://github.com/tteck/Proxmox ### Updates Repository For regular updates and to avoid errors set the updates repository from the enterprise proxmox repo (subscription required) to the `pve-no-subscription` repo. ![[2022-01-25-23-37-43.png]] 0. `pve node > updates > repositories` 1. disable the enterprise repo 2. `[Add]` 3. `pve-no-subscription` 4. run `apt-get update; apt dist-upgrade; reboot` ### Enable IOMMU Enable [[Input-Output Memory Management Unit|iommu]] so VM's can access hardware not made for virtualization (GPU's etc.) 1. you can do this but updating the `/etc/default/grub` file - change `GRUB_CMDLINE_LINUX_ DEFAULT="quiet"` - to: `GRUB_CMDLINE LINUX DEFAULT="quiet intel iommu=on"` 2. Then run `update-grub` 3. Then edit `/etc/modules` Add these 4 lines to it: 4. ```txt `vfio` `vfio_iommu_typel` `vfio_pci` `vfio_virqfd` ``` 5. Then run `update-initramfs -u -k all` 6. reboot ### Make Proxmox VLAN aware 1. go to `pve node > System > Network` 2. "Edit" your Linux bridge 3. check the box for `VLAN aware:` 4. Click `Apply Configuration` This will update `/etc/network/interfaces` with new settings and where it says `bridge-vids` you can change the default `2-4094` to be a single number for the [[Virtual Local Area Network|VLAN]] of the server, or do that for individual virtual machines ### Setup Linux Bridge for Virtual Machines Separate from management Layer 1. `pve node > System > Network > Create > Linux Bond` 2. `bond0` 3. List all the bridge ports in a space separated list except the 1 used for the management layer 4. choose `ieee-802.3ad` mode for [[LACP]] 5. Add Comment 6. after finished creating modify switch side settings for [[LACP]] for those ports ##### Make Network Bridge for Virtual Machines - <https://youtu.be/qTbeHpdHcqs> 1. `pve node > System > Network > Create > Linux Bridge` 2. `vmbr1` is fine 3. Give it a IPV4 address like `10.10.10.0/24` 4. make it `VLAN aware:` 5. List all the bridge ports in a space separated list (the [[LACP]] `bond0` you made) 6. Add Comment ### Setup NFS for backups 0. You need to have the NFS share already setup so [[Fafnir]] needs to already be setup and mounted to the proxmox instance? 1. `Datacenter node > storage > add > nfs` 2. `ID` ==> "Backups" 3. Server IPV4 address (address to [[Fafnir]]?) 4. Export `/mnt/storage <++>` ##### Schedule Backups 1. `Datacenter node > backup > add` 2. Select Node to backup 3. Select storage share to send backups to 4. Schedule Backups 5. Email notification Settings 6. Compression level (ZSTD) 7. mode == snapshot 8. test it 1. make a backup immediately ### Download Windows VirtIO drivers 1. Go To [This page](https://pve.proxmox.com/wiki/Windows_VirtIO_Drivers) 2. Click the link under `Installation` for downloading latest stable release 3. upload iso to proxmox iso's in `local > ISO images > Upload` ### Configure Email notifications Change `/etc/postfix/main.cf` to include/change these lines: ```txt relayhost = [smtp.gmail.com]:587 smtp_use_tls = yes smtp_sasl_auth_enable = yes smtp_sasl_security_options = noanonymous smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt #mydestination = $myhostname, localhost.$mydomain, localhost ``` Be sure there are no dupes as the `main.cf` may have `smtp_sasl_security_options = {}` , and `relayhost = {}`. Just delete or comment those lines. Create an `/etc/postfix/sasl_passwd` file with: ```txt [smtp.gmail.com]:587 [email protected]:PASSWD ``` run ```bash chmod 600 /etc/postfix/sasl_passwd postmap /etc/postfix/sasl_passwd ``` install for passwd support: ```bash apt-get install libsasl2-modules ``` Restart service: ```bash systemctl restart postfix.service ``` Test: ```bash echo "Test mail from postfix" | mail -s "Test Postfix" [email protected] ``` Test from PVE: ```bash echo "test" | /usr/bin/pvemailforward ``` ### Setup port forwarding for RDP to windows VM's and make VM's visible on the internal network: 1. Start a shell from the web console 2. edit `/etc/network/interfaces` 3. make it look like: ```bash auto vmbr1 iface vmbr1 inet static address 10.1.10.0/24 bridge-ports bond0 bridge-stp off bridge-fd 0 bridge-vlan-aware yes bridge-vids 20 post-up echo 1 > /proc/sys/net/ipv4/ip_forward post-up iptables -t nat -A POSTROUTING -s '192.168.0.0/24' -o vmbr0 -j MASQUERADE post-down iptables -t nat -D POSTROUTING -s '192.168.0.0/24' -o vmbr0 -j MASQUERADE iptables -t nat -A PREROUTING -i bond0 -p tcp --dport 13389 -j DNAT --to 192.168.3.15:3389 #VM Net ``` ### Setup iperf3 on the server ```bash apt-get install iperf3 ``` In the `~/.profile` file, add this line: ```bash iperf3 -s & ``` this will make it so upon server startup iperf3 will be run as an independant process that can can gather data from. By default it listens on port `5021` ## Tips & Tricks In the Server view, the `Datacenter` option is for settings that affect all your nodes. In essence the "tree" view allows setting editing for all children under the parent item. ### VM Templates When you have a default VM config you like, power down the VM, right click, Convert to template. It can no longer be used as a VM but you can clone from that template - Mode: - Linked clone: dependent child on the template with cascading downstream changes - Full Clone: independent entity - After cloning these steps will reset important information - Change Hostname - `sudo vim /etc/hostname` - Change hosts file - `sudo vim /etc/hosts` - Reset Machine ID - ```bash rm -f /etc/machine-id /var/lib/dbus/machine-id dbus-uuidgen --ensure=/etc/machine-id dbus-uuidgen --ensure ``` - Regenerate ssh keys - ```bash regen ssh keys sudo rm /etc/ssh/ssh_host_* sudo dpkg-reconfigure openssh-server ```